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Chapter 


Linear Models 


Each chapter in this manual normally has its own statistical background section. In 
this part, however, Regression, ANOVA, and General Linear Models are grouped 
together. There are two reasons for doing this. First, while some introductory 
textbooks treat regression and analysis of variance as distinct, statisticians know that 
they are based on the same underlying mathematical model. When you study what 
these procedures do, therefore, it is helpful to understand this model and learn the 
common terminology underlying each method. Second, although SYSTAT has three 
commands (REGRESS, ANOVA, and GLM) and menu settings, it is a not-so-well- 
guarded secret that all these lead to the same program, originally called MGLH (for 
Multivariate General Linear Hypothesis). Having them organized this way means that 
SYSTAT can use tools designed for one approach (for example, dummy variables in 
ANOVA) in another (such as computing within-group correlations in multivariate 
regression). This synergy is not usually available in packages that treat these models 
independently. 


Simple Linear Models 


Linear models are models based on /ines. More generally, they are based on linear 
surfaces, such as lines, planes, and hyperplanes. Linear models are widely applied 
because lines and planes often appear to describe well the relations among variables 
measured in the real world. We will begin by examining the equation for a straight 
line, and then move to more complex linear models. 


Ir-1 


1-2 


Chapter 1 


Equation for a Line 


A linear model looks like this: 
y =a+bx 


This is the equation for a straight line that you learned in school. The quantities in this 
equation are: 


y  adependent variable 
x an independent variable 


Variables are quantities that can vary (have different numerical values) in the same 
equation. The remaining quantities are called parameters. A parameter is a quantity 
that is constant in a particular equation, but that can be varied to produce other 
equations in the same general family. The parameters are: 


a The value of y when x is 0. This is sometimes called a y-intercept (where a line intersects 
the y axis in a graph when x is 0). 


b The slope of the line, or the number of units y changes when x changes by one unit. 


Let us look at an example. Here are some data showing the yearly earnings a partner 
should theoretically get in a certain large law firm, based on annual personal billings 
over quota (both in thousands of dollars): 


EARNINGS BILLINGS 


60 20 
70 40 
80 60 
90 80 
100 100 
120 140 
140 180 
150 200 
175 250 
190 280 


We can plot these data with EARNINGS on the vertical axis (dependent variable) and 
BILLINGS on the horizontal (independent variable). Notice in the following figure that 
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all the points lie on a straight line. 


3004 
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What is the equation for this line? Look at the vertical axis value on the sloped line 
where the independent variable has a value of 0. Its value is 50. A lawyer is paid 
$50,000 even when billing nothing. Thus, a is 50 in our equation. What is b? Notice 
that the line rises by $10,000 when billings change by $20,000. The line rises half as 
fast as it runs. You can also look at the data and see that the earnings change by $1 as 
billing changes by $2. Thus, b is 0.5, ora half, in our equation. 

Why bother with all these calculations? We could use the table to determine a 
lawyer's compensation, but the formula and the line graph allow us to determine wages 
not found in the table. For example, we now know that $30,000 in billings would yield 
earnings of $65,000: 


EARNINGS = 50000 + 0.5 x 30000 = 65000 


When we do this, however, we must be sure that we can use the same equation on these 
new values. We must be careful when interpolating, or estimating, wages for billings 
between the ones we have been given. Does it make sense to compute earnings for 
$25,000 in billings, for example? It probably does. Similarly, we must be careful when 
extrapolating, or estimating from units outside the domain of values we have been 
given. What about negative billings, for example? Would we want to pay an 
embezzler? Be careful. Equations and graphs usually are meaningful only within or 
close to the range of y values and domain of x values in the data. 
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Regression 


Data are seldom this clean unless we design them to be that way. Law firms typically 
fine tune their partners’ earnings according to many factors. Here are the real billings 
and earnings for our law firm (these lawyers predate Reagan, Bush, Clinton, and 


Gates): 

EARNINGS BILLINGS 
86 20 
67 40 
95 60 
105 80 
86 100 
82 140 
140 180 
145 200 
144 250 
184 280 


Our techniques for computing a linear equation won't work with these data. Look at 
the following graph. There is no way to draw a straight line through all the data. 


EARNINGS 


0 100 200 300 
BILLINGS 


Given the irregularities in our data, the line drawn in the figure is a compromise. How 
do we find a best fitting line? If we are interested in predicting earnings from the billing 
data values rather well, a reasonable method would be to place a line through the points 
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so that the vertical deviations between the points and the line (errors in predicting 
earnings) are as small as possible. In other words, these deviations (absolute 
discrepancies, or residuals) should be small, on the average, for a good-fitting line. 

The procedure of fitting a line or curve to data such that residuals on the dependent 
variable are minimized in some way is called regression. Because we are minimizing 
vertical deviations, the regression line often appears to be more horizontal than we 
might place it by eye, especially when the points are fairly scattered. It "regresses" 
toward the mean value of y across all the values of x, namely, a horizontal line through 
the middle of all the points. The regression line is not intended to pass through as many 
points as possible. It is for predicting the dependent variable as accurately as possible, 
given each value of the independent variable. 


Least Squares 


There are several ways to draw the line so that, on the average, the deviations are small. 
We could minimize the mean, the median, or some other measure of the typical 
behavior of the absolute values of the residuals. Or we can minimize the sum (or mean) 
of the squared residuals, which yields almost the same line in most cases. Using 
squared instead of absolute residuals gives more influence to points whose y value is 
farther from the average of all y values. This is not always desirable, but it makes the 
mathematics simpler. This method is called ordinary least squares. 

By specifying EARNINGS as the dependent variable and BILLINGS as the 
independent variable in a MODEL statement, we can compute the ordinary least- 
squares regression y-intercept as $62,800 and the slope as 0.375. These values do not 
predict any single lawyer’s earnings exactly. They describe the whole firm well, in the 
sense that, on the average, the line predicts a given earnings value fairly closely from 
a given billings value. 


Estimation and Inference 


We often want to do more with such data than draw a line on a picture. In order to 
generalize, formulate a policy, or test a hypothesis, we need to make an inference. 
Making an inference implies that we think a model describes a more general 
population from which our data have been randomly sampled. In the present example, 
this population is all possible lawyers who might work for this firm. To make an 
inference about compensation, we need to construct a linear model for our population 
that includes a parameter for random error. In addition, we need to change our notation 
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to avoid confusion later. We are going to use Greek letters to denote parameters and 
italic Roman letters for variables. The error parameter is usually called e. 


y=at+Pxte 


Notice that e is a random variable. It varies like any other variable (for example, x), 
but it varies randomly, like the tossing of a coin. Since e is random, our model forces 
y to be random as well because adding fixed values (a and Bx ) to a random variable 
produces another random variable. In ordinary language, we are saying with our model 
that earnings are only partly predictable from billings. They vary slightly according to 
many other factors, which we assume are random. 

We do not know all of the factors governing the firm’s compensation decisions, but 
we assume: 


m All the salaries are derived from the same linear model. 


= The error in predicting a particular salary from billings using the model is 
independent of (not in any way predictable from) the error in predicting other 
salaries. 


m The errors in predicting all the salaries come from the same random distribution. 


Our model for predicting in our population contains parameters, but unlike our perfect 
straight line example, we cannot compute these parameters directly from the data. The 
data we have are only a small sample from a much larger population, so we can only 
estimate the parameter values using some statistical method on our sample data. Those 
of you who have heard this story before may not be surprised that ordinary least 
squares is one reasonable method for estimating parameters when our three 
assumptions are appropriate. Without going into all the details, we can be reasonably 
assured that if our population assumptions are true and if we randomly sample some 
cases (that is, each case has an equal chance of being picked) from the population, the 
least-squares estimates of o. and B will, on the average, be close to their values in the 
population. 

So far, we have done what seems like a sleight of hand. We delved into some 
abstruse language and came up with the same least-squares values for the slope and 
intercept as before. There is something new, however. We have now added conditions 
that define our least-squares values as sample estimates of population values. We now 
regard our sample data as one instance of many possible samples. Our compensation 
model is like Plato's cave metaphor; we think it typifies how this law firm makes 
compensation decisions about any lawyer, not just the ones we sampled. Before, we 
were computing descriptive statistics about a sample. Now, we are computing 
inferential statistics about a population. 
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Standard Errors 


There are several statistics relevant to the estimation of œ and f . Perhaps most 
important is a measure of how variable we could expect our estimates to be if we 
continued to sample data from our population and used least squares to get our 
estimates. A statistic calculated by SYSTAT shows what we could expect this variation 
to be. It is called, appropriately, the standard error of estimate, or Std Error in the 
output. The standard error of the y-intercept, or regression constant, is in the first row 
of the coefficients: 10.440. The standard error of the billing coefficient or slope is 
0.065. Look for these numbers in the following output: 


Dependent Variable | EARNINGS 
N | 10 
Multiple R 0.897 
Squared Multiple R 0.804 
Adjusted Squared Multiple R } 0 719 
Standard Error of Estimate | 17.626 


Regression Coefficients B = (x'x) dx Y 


Effect | Coefficient Tolerance t p-value 
precast lar A WS PE casalis Let cae ci alo it 
CONSTANT | 62.838 . 6.019 0.000 
BILLINGS | 0.375 1.000 5.728 0.000 


Analysis of Variance 


Source i ss df Mean Squares 
pri AN A eee nenenn ere seenen 
Regression | 10191.109 1 10191.109 
Residual |! 2485.291 8 310.661 


Hypothesis Testing 


From these standard errors, we can construct hypothesis tests on these coefficients. 
Suppose a skeptic approached us and said, “Your estimates look as if something is 
going on here, but in this firm, salaries have nothing to do with billings. You just 
happened to pick a sample that gives the impression that billings matter. It was the luck 
of the draw that provided you with such a misleading picture. In reality, B. is 0 in the 
population because billings play no role in determining earnings." 

We can reply, “Jf salaries had nothing to do with billings but are really just a mean 
value plus random error for any billing level, then would it be likely for us to find a 
coefficient estimate for f at least this different from 0 in a sample of 10 lawyers?" 

To represent these alternatives as a bet between us and the skeptic, we must agree 
on some critical level for deciding who will win the bet. If the likelihood of a sample 
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result at least this extreme occurring by chance is less than or equal to this critical level 
(say, five times out of a hundred), we win; otherwise, the skeptic wins. 

This logic might seem odd at first because, in almost every case, our skeptic’s null 
hypothesis would appear ridiculous, and our alternative hypothesis (that the skeptic is 
wrong) seems plausible. Two scenarios are relevant here, however. The first is the 
lawyer’s. We are trying to make a case here. The only way we will prevail is if we 
convince our skeptical jury beyond a reasonable doubt. In statistical practice, that 
reasonable doubt level is relatively liberal: fewer than five times in a hundred. The 
second scenario is the scientist's. We are going to stake our reputation on our model. 
If someone sampled new data and failed to find nonzero coefficients, much less 
coefficients similar to ours, few would pay attention to us in the future. 

To compute probabilities, we must count all possibilities or refer to a mathematical 
probability distribution that approximates these possibilities well. The most widely 
used approximation is the normal curve, which we reviewed briefly in Chapter 1 in 
Statistics I. For large samples, the regression coefficients will tend to be normally 
distributed under the assumptions we made above. To allow for smaller samples, 
however, we will add the following condition to our list of assumptions: 


= The errors in predicting the salaries come from a normal distribution. 


If we estimate the standard errors of the regression coefficients from the data instead 
of knowing them in advance, then we should use the / distribution instead of the 
normal. The two-tail value for the probability represents the area under the theoretical 
t probability curve corresponding to coefficient estimates whose absolute values are 
more extreme than the ones we obtained. For both parameters in the model of lawyers’ 
earnings, these values (given as p-value(2 tail)) are less than 0.001, leading us to reject 
our null hypothesis at well below the 0.05 level. 

At the bottom of our output, we get an analysis of variance table that tests the 
goodness of fit of our entire model. The null hypothesis corresponding to the F-ratio 
(32.805) and its associated p-value is that the billing variable coefficient is equal to 0. 
This test overwhelmingly rejects the null hypothesis that both o. and B are 0. 


Multiple Correlation 


In the same output is a statistic called the Squared multiple correlation. This is the 

proportion of the total variation in the dependent variable (EARNINGS) accounted for by 
the linear prediction using BILLINGS. The value here (0.804) tells us that approximately 
80% of the variation in earnings can be accounted for by a linear prediction from billings. 
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The rest of the variation, as far as this model is concerned, is random error. The square 
root of this statistic is called, not surprisingly, the multiple correlation. The adjusted 
squared multiple correlation (0.779) is what we would expect the squared multiple 
correlation to be if we used the model we just estimated on a new sample of 10 lawyers 
in the firm. It is smaller than the squared multiple correlation because the coefficients 
were optimized for this sample rather than for the new one. 


Regression Diagnostics 


We do not need to understand the mathematics of how a line is fitted in order to use 
regression. You can fit a line to any x-y data by the method of least-squares. The 
computer doesn’t care where the numbers come from. To have a model and estimates 
that mean something, however, you should be sure the assumptions are reasonable and 
that the sample data appear to be sampled from a population that meets the 


assumptions. 
The sample analogues of the errors in the population model are the residuals—the 


differences between the observed and predicted values of the dependent variable. 
There are many diagnostics you can perform on the residuals. Here are the most 


important ones: 
The errors are normally distributed. Draw a normal probability plot (PPLOT) of the 
residuals. 


o 
T 
. 


Expected Value for Normal Distribution 
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The residuals should fall approximately on a diagonal straight line in this plot. When 
the sample size is small, as in our law example, the line may be quite jagged. It is 
difficult to tell by any method whether a small sample is from a normal population. You 
can also plot a histogram or stem-and-leaf diagram of the residuals to see if they are 
lumpy in the middle with thin, symmetric tails. 


The errors have constant variance. Plot the residuals against the estimated values. The 
following plot shows studentized residuals (STUDENT) against estimated values 
(ESTIMATE). Studentized residuals are the true “external” kind discussed in Velleman 
and Welsch (1981). Use these statistics to identify outliers in the dependent variable 
space. Under normal regression assumptions, they have a z distribution with 

(N-p- 1) degrees of freedom, where N is the total sample size and p is the number 
of predictors (including the constant). Large values (greater than 2 or 3 in absolute 
magnitude) indicate possible problems. 
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Our residuals should be arranged in a horizontal band within two or three units around 
0 in this plot. Again, since there are so few observations, it is difficult to tell whether 
they violate this assumption in this case. There is only one particularly large residual, 
and it is toward the middle of the values. This lawyer billed $140,000 and is earning 
only $80,000. He or she might have a gripe about supporting a higher share of the 
firm’s overhead. 


The errors are independent. Several plots can be done. Look at the plot of residuals 
against estimated values above. Make sure that the residuals are randomly scattered 
above and below the 0 horizontal and that they do not track in a snaky way across the 
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plot. If they look as if they were shot at the plot by a horizontally moving machine gun, 
then they are probably not independent of each other. You may also want to plot 
residuals against other variables, such as time, orientation, or other ways that might 
influence the variability of your dependent measure. ACF PLOT in SERIES measures 
whether the residuals are serially correlated. Here is an autocorrelation plot: 


Autocorrelation Plot 
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All the bars should be within the confidence bands if each residual is not predictable 
from the one preceding it, and the one preceding that, and the one preceding that, and 


so on. 


All the members of the population are described by the same linear model. Plot 
Cook's distance (COOK) against the estimated values. 
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Cook's distance measures the influence of each sample observation on the coefficient 
estimates. Observations that are far from the average of all the independent variable 
values or that have large residuals tend to have a large Cook's distance value (say, 
greater than 2). Cook's D actually closely follows an F distribution, so aberrant values 
depend on the sample size. As a rule of thumb, under the normal regression 
assumptions, COOK can be compared to an F distribution with p and N — p degrees of 
freedom. We don't want to find a large Cook's D value for an observation because it 
would mean that the coefficient estimates would change substantially if we deleted that 
observation. While none of the COOK values are extremely large in our example, could 
it be that the largest one in the upper right corner is the founding partner in the firm? 
Despite large billings, this partner is earning more than the model predicts. 

Another diagnostic statistic useful for assessing the model fit is leverage, discussed 
in Belsley, Kuh, and Welsch (1980) and Velleman and Welsch (1981). Leverage helps 
to identify outliers in the independent variable space. Leverage has an average value 
of p/N, where p is the number of estimated parameters (including the constant) and 
Nis the number of cases. What is a high value of leverage? In practice, it is useful to 
examine the values in a stem-and-leaf plot and identify those that stand apart from the 
rest of the sample. However, various rules of thumb have been suggested. For example, 
values of leverage less than 0.2 appear to be safe; between 0.2 and 0.5, risky; and above 
0.5, to be avoided. Another says that if p > 6 and (N — p) > 12, use (3p)/N asa cutoff. 
SYSTAT uses an F approximation to determine this value for warnings (Belsley, Kuh, 
and Welsch, 1980). 

In conclusion, keep in mind that all our diagnostic tests are themselves a form of 
inference. We can assess theoretical errors only through the dark mirror of our 
observed residuals. Despite this caveat, testing assumptions graphically is critically 
important. You should never publish regression results until you have examined these 
plots. 


Multiple Regression 
A multiple linear model has more than one independent variable; that is: 


y=atbx+cz 


This is the equation for a plane in three-dimensional space. The parameter a is still an 
intercept term. It is the value of y when x and z are 0. The parameters b and c are still 
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slopes. One gives the slope of the plane along the x dimension; the other, along the 
z dimension. 


The statistical model has the same form: 
y=atPxtyzte 


Before we run out of letters for independent variables, let us switch to a more 
frequently used notation: 


y = Bot Bix, + Bx +E 


Notice that we are still using Greek letters for unobservables and Roman letters for 
observables. 

Now, let us look at our law firm data again. We have learned that there is another 
variable that appears to determine earnings—the number of hours billed per year by 
each lawyer. Here is an expanded listing of the data: 


EARNINGS BILLINGS HOURS 


86 20 1771 
67 40 1556 
95 60 1749 
105 80 1754 
86 100 1594 
82 140 1400 
140 180 1780 
145 200 1737 
144 250 1645 
184 280 1863 


For our model, f is the coefficient for BILLINGS, and is the coefficient for HOURS. 
Let us look first at its graphical representation. The following figure shows the plane 
fit by least-squares to the points representing each lawyer. Notice how the plane slopes 
upward on both variables. BILLINGS and HOURS both contribute positively to 
EARNINGS in our sample. 
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Fitted Model Plot 


Fitting this model involves no more work than fitting the simple regression model. We 
specify one dependent and two independent variables and estimate the model as 
before. Here is the result: 


Dependent Variable 
N 


Multiple R 

Squared Multiple R 

Adjusted Squared Multiple R 
Standard Error of Estimate 


Regression Coefficients B = (X'X)x'Y 
Std. 


w 
m 
bj 
o 
a 
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Coefficient Standard Error Coefficient 
CONSTANT -139.925 
BILLINGS 0.333 . 
HOURS 0.124 0.007 0.449 


Analysis of Variance 


Source Mean Squares F-ratio p-value 


Regression | 12626.210 2 6313.105 880.493 0.000 
Residual i 50.190 7 7.170 
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This time, we have one more row in our regression table for HOURS. Notice that its 
coefficient (0.124) is smaller than that for BILLINGS (0.333). This is due partly to the 
different scales of the variables. HOURS are measured in larger numbers than 
BILLINGS. If we wish to compare the influence of each, independent of scales, we 
should look at the standardized coefficients. Here, we still see that BILLINGS (0.797) 
play a greater role in predicting EARNINGS than do HOURS (0.449). Notice also that 
both coefficients are highly significant and that our overall model is highly significant, 
as shown in the analysis of variance table. 


Variable Selection 


In applications, you may not know which subset of predictor variables in a larger set 
constitutes a “good” model. Strategies for identifying a good subset are many and 
varied: forward selection, backward elimination, stepwise (either a forward or 
backward type), and all subsets. Forward selection begins with the “best” predictor, 
adds the next “best” and continues entering variables to improve the fit. Backward 
selection begins with all candidate predictors in an equation and removes the least 
useful one at a time as long as the fit is not substantially “worsened.” Stepwise begins 
as either forward or backward, but allows “poor” predictors to be removed from the 
candidate model or “good” predictors to re-enter the model at any step. Finally, all 
subsets methods compute all possible subsets of predictors for each model of a given 
size (number of predictors) and choose the “best” one. 


Bias and variance tradeoff. Submodel selection is a tradeoff between bias and 
variance. By decreasing the number of parameters in the model, its predictive 
capability is enhanced. This is because the variance of the parameter estimates 
decreases. On the other side, bias may increase because the “true model” may have a 
higher dimension. So we’d like to balance smaller variance against increased bias. 
There are two aspects to variable selection: selecting the dimensionality of the 
submodel (how many variables to include) and evaluating the model selected. After 
you determine the dimension, there may be several alternative subsets that perform 
equally well. Then, knowledge of the subject matter, how accurately individual 
variables are measured, and what a variable “communicates” may guide the selection 
of the model to report. 

A strategy. If you are in an exploratory phase of research, you might try this version of 
backwards stepping. First, fit a model using all candidate predictors. Then identify the 
least “useful” variable, remove it from the model list, and fit a smaller model. Evaluate 
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your results and select another variable to remove. Continue removing variables. For 
a given size model, you may want to remove alternative variables (that is, first remove 
variable A, evaluate results, replace 4 and remove B, etc.). 


Entry and removal criteria. Decisions about which variable to enter or remove should 
be based on statistics and diagnostics in the output, especially graphical displays of 
these values, and your knowledge of the problem at hand. 

You can specify your own alpha-to-enter and alpha-to-remove values (do not make 
alpha-to-remove less than alpha-to-enter), or you can cycle variables in and out of the 
equation (stepping automatically stops if this happens). The default values for these 
options are Enter = 0.15 and Remove = 0.15. These values are appropriate for predictor 
variables that are relatively independent. If your predictor variables are highly 
correlated, you should consider lowering the Enter and Remove values well below 
0.05. 

When there are high correlations among the independent variables, the estimates of 
the regression coefficients can become unstable. Tolerance is a measure of this 
condition. It is (1 — A^) ; that is, one minus the squared multiple correlation between a 
predictor and the other predictors included in the model, (Note that the dependent 
variable is not used.) By setting a minimum tolerance value, variables highly correlated 
with others already in the model are not allowed to enter. 

As a rough guideline, consider models that include only variables that have absolute 
t values well above 2.0 and “tolerance” values greater than 0.1. (We use quotation 
marks here because ¢ and other statistics do not have their usual distributions when you 
are selecting subset models.) 


Evaluation criteria. There is no one test to identify the dimensionality of the best 
submodel. Research by Leo Breiman emphasizes the usefulness of cross-validation 
techniques involving 80% random subsamples. Sample 80% of your file, fit a model, 
use the resulting coefficients on the remaining 20% to obtain predicted values, and then 
compute R° for this smaller sample. In over-fitting situations, the discrepancy between 
the R^ for the 80% sample and the 20% sample can be dramatic, 


A warning. If you do not have extensive knowledge of your variables and expect this 
strategy to help you to find a “true” model, you can get into a lot of trouble. Automatic 
stepwise regression programs cannot do your work for you. You must be able to 
examine graphics and make intelligent choices based on theory and prior knowledge: 
otherwise, you will be arriving at nonsense. 

Moreover, if you are thinking of testing hypotheses after automatically fitting a 
subset model, don’t bother. Stepwise regression programs are the most notorious 
source of “pseudo” p-values in the field of automated data analysis. Statisticians seem 
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to be the only ones who know these are not “real” p-values. The automatic stepwise 
option is provided to select a subset model for prediction purposes. It should never be 
used without cross-validation. 

If you still want some sort of confidence estimate on your subset model, you might 
look at tables in Wilkinson (1979), Rencher and Pun (1980), and Wilkinson and Dallal 
(1982). These tables provide null hypothesis R? values for selected subsets given the 
number of candidate predictors and final subset size. If you don’t know this literature 
already, you will be surprised at how large multiple correlations from stepwise 
regressions on random data can be. For a general summary of these and other 
problems, see Hocking (1983). For more specific discussions of variable selection 
problems, see the previous references and Flack and Chang (1987), Freedman (1983), 
and Lovell (1983). Stepwise regression is probably the most abused computerized 
statistical technique ever devised. If you think you need automated stepwise regression 
to solve a particular problem, it is almost certain that you do not. Professional 
statisticians rarely use automated stepwise regression because it does not necessarily 
find the “best” fitting model, the “real” model, or alternative “plausible” models. 
Furthermore, the order in which variables enter or leave a stepwise program is usually 
of no theoretical significance. You are always better off thinking about why a model 
could generate your data and then testing that model, AIC and Schwarz’s BIC. Model 
selection criteria like likelihood and multiple- R? are biased towards models with more 
parameters, leading to over-fitted and less precise models. 

Akaike (1973, 1974), proposed the Akaike Information Criterion (AIC) as a model 
selection criterion as follows: 

AIC= -2Log-likelihood +2k, where k is the number of parameters estimated. 

Model selection using AIC is based on the principle of parsimony. AIC penalizes 
the likelihood with respect to the number of parameters estimated.The AIC value of a 
model can be interpreted as an estimate of the relative discrepancy between the model 
and the unknown true model which generated the data. The idea of model selection 
using AIC is to select a model with a low AIC value. Model selection using AIC is 
asympotically equivalent to model selection by cross-validation. Proper care should be 
taken for model selection using AIC, and the AIC values should be used to compare 
models based on the same data and the same response. AIC may perform poorly if the 
number of parameters estimated is more relative to the number of observations. 

Hurvich and Tsai (1989), provided small sample Akaike information criterion 
called, AIC (corrected) as follows; 

AIC (corrected) = -2Log-likelihood+2k+2k (k+1)/(n-k-1), where n is the number of 
observations. AIC (corrected) is applicable only for linear models with the underlying 
distribution being Gaussian. 
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Schwarz (1978) provided a Bayesian Information Criterion (BIC) for model 
selection. 

Schwarz’s BIC = -2*Log- likelihood + k*log(n). Schwarz’s BIC value of a model 
can also be interpreted as an estimate of relative discrepancy between the model and 
the unknown true model which generated the data. The idea is to select a model with a 
low Schwarz’s BIC value. 

Burnham and Anderson (2003) is a good source of material on information criteria 
and model selection. 

In linear regression, ANOVA, GLM, and MANOVA the Log-likelihood is obtained 
under the assumption of normality. 

In SYSTAT AIC, AIC (corrected) and Schwarz’s BIC values are provided in Linear 
Regression (Least-Squares), ANOVA, GLM, Logit Regression, Probit Regression, 
Survival Analysis and MANOVA features. In Linear regression, ANOVA, GLM, and 
MANOVA the log-likelihood is obtained under the assumption of normality. 


Using an SSCP, a Covariance, or a 
Correlation Matrix as Input 


Normally for a regression analysis, you use a cases-by-variables data file. You can, 
however, use a covariance or correlation matrix saved (from Correlations) as input. If 
you use a matrix as input, specify the sample size that generated the matrix where the 
number you type is an integer greater than two. 

You can enter an SSCP, a covariance, or a correlation matrix by typing it into the 
Data Editor Worksheet, by using BASIC, or by saving it ina SYSTAT file. Be sure to 
include the dependent as well as independent variables. 

SYSTAT needs the sample size to calculate degrees of freedom, so you need to 
enter the original sample size. Least-Squares determines the type of matrix (SSCP, 
covariance, etc.) and adjusts appropriately. With a correlation matrix, the raw and 
standardized coefficients are the same. Therefore, the Include constant option is 
disabled when using SSCP, covariance, or correlation matrices. Because these 
matrices are centered, the constant term has already been removed. 

The following two analyses of the same data file produce identical results (except 
that you don’t get residuals with the second). In the first, we use the usual cases-by- 
variables data file. In the second, we use the CORR command to save a covariance 
matrix and then analyze that matrix file with the REGRESS command. 
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Here are the usual instructions for a regression analysis: 


REGRESS 
USE FILENAME 
MODEL Y = CONSTANT + X(1) + X(2) + X(3) 
ESTIMATE 


Here, we compute a covariance matrix and use itin the regression analysis: 


CORR 
USE FILENAME1 
SAVE filename2 
COVARIANCE X(1) X(2) X(3) Y 


REGRESS 
USE FILENAME2 
MODEL Y = X(1) + X(2) + X(3) / N=40 
ESTIMATE 


The triangular matrix input facility is useful for “meta-analysis” of published data and 
missing-value computations. There are a few warnings, however. First, if you input 
correlation matrices from textbooks or articles, you may not get the same regression 
coefficients as those printed in the source. Because of round-off error, printed and raw 
data can lead to different results. Second, if you use pairwise deletion with CORR, the 
degrees of freedom for hypotheses will not be appropriate. You may not even be able 
to estimate the regression coefficients because of singularities. 

In general, when an incomplete data procedure is used to estimate the correlation 
matrix, the estimate of regression coefficients and hypothesis tests produced from itare 
optimistic. You can correct for this by specifying a sample size smaller than the 
number of actual observations (preferably, set it equal to the smallest number of cases 
used for any pair of variables), but this is a crude guess that you could refine only by 
doing Monte Carlo simulations. There is no simple solution. Beware, especially, of 
multivariate regressions (or MANOVA, etc.) with missing data on the dependent 
variables. You can usually compute coefficients, but results from hypothesis tests are 


particularly suspect. 


Analysis of Variance 


Often, you will want to examine the influence of categorical variables (such as gender, 
species, country, and experimental group) on continuous variables. The model 
equations for this case, called analysis of variance, are equivalent to those used in 
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linear regression. However, in the latter, you have to figure out a numerical coding for 
categories so that you can use the codes in an equation as the independent variable(s). 


Effects Coding 


The following data file, EARNBILL, shows the breakdown of lawyers sampled by sex. 
Because SEX is a categorical variable (numerical values assigned to MALE or 
FEMALE are arbitrary), a code variable with the values 1 or —1 is used. It doesn't 
matter which group is assigned —1, as long as the other is assigned 1. 


EARNINGS SEX CODE 
86 female -1 
67 female -1 
95 female -1 
105 female -1 
86 female -1 
82 male 

140 male 1 
145 male 1 
144 male 1 
184 male 1 


There is nothing wrong with plotting earnings against the code variable, as long as you 
realize that the slope of the line is arbitrary because it depends on how you assign your 
codes. By changing the values of the code variable, you can change the slope. Here is 
a plot with the least-squares regression line superimposed. 
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EARNINGS 


1 
CODE 


Let us do a regression on the data using these codes. Here are the coefficients as 
computed by ANOVA: 


Variable Coefficients 


Constant 113.400 
Code 25.600 


Notice that Constant (113.4) is the mean of all the data. It is also the regression 
intercept because the codes are symmetrical about 0. The coefficient for Code (25.6) 
is the slope of the line. It is also one half the difference between the means of the 
groups. This is because the codes are exactly two units apart. This slope is often called 
an effect in the analysis of variance because it represents the amount that the 
categorical variable SEX affects BILLINGS. In other words, the effect of SEX can be 
represented by the amount that the mean for males differs from the overall mean. 


Means Coding 


The effects coding model is useful because the parameters (constant and slope) can be 
interpreted as an overall level and as the effect(s) of treatment, respectively. Another 


momma re 
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Models 


model, however, that yields the means of the groups directly is called the means model. 
Here are the codes for this model:. 


EARNINGS SEX . CODEI CODE2 


86 female 1 0 
67 female 1 0 
95 female 1 0 
105 female 1 0 
86 female 1 0 
82 male 0 1 
140 male 0 1 
145 male 0 1 
144 male 0 1 
184 male 0 1 


Notice that CODE] is nonzero for all females, and CODE? is nonzero for all males. To 

estimate a regression model with these codes, you must leave out the constant. With 

only two groups, only two distinct pieces of information are needed to distinguish 

them. Here are the coefficients for these codes in a model without a constant: 
Variable Coefficient 


Codel 87.800 
Code2 139.000 


Notice that the coefficients are now the means of the groups. 


Let us look at the algebraic models for each of these codings. Recall that the regression 
model looks like this: 


y = Bot Bix, +e 
For the effects model, it is convenient to modify this notation as follows: 
Yyrpta te 


~ 

When x (the code variable) is —1, o; is equivalent to a,; when x is 1, a, is equivalent to 
05. This shorthand will help you later when dealing with models with many categories. 
For this model, the i parameter stands for the grand (overall) mean, and the a 


$ 
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parameter stands for the effect. In this model, our best prediction of the score ofa group 
member is derived from the grand mean plus or minus the deviation of that group from 
this grand mean. 


The means model looks like this: 


yj7Wwtse 


In this model, our best prediction of the score of a group member is the mean of that 
group. It's noteworthy that, whatever be the coding (means, effect, or dummy) the 
prediction of the score of a group member remains the same. 


Hypotheses 


As with regression, we are usually interested in testing hypotheses concerning the 
parameters of the model. Here are the hypotheses for the two models: 


Ho: a= a= 0 (effects model) 
Ho: p; = pz (means model) 


The tests of this hypothesis compare variation between the means to variation within 
each group, which is mathematically equivalent to testing the significance of 
coefficients in the regression model. In our example, the F-ratio in the analysis of 
variance table tells you that the coefficient for SEX is significant at p — 0.019, which is 
less than the conventional 0.05 value. Thus, on the basis of this sample and the validity 
of our usual regression assumptions, you can conclude that women earn significantly 
less than men in this firm. 

Dependent Variable ! sg 


N 

Multiple R | 0.719 
Squared Multiple R | 0.517 
Analysis of Variance 


Source | Type III SS df Mean Squares F-ratio p-value 


parnana A A ER 
SEX$ i 6553.600 1 6553.600 8.563 0.019 


Error | 6122.800 8 765.350 


The nice thing about realizing that ANOVA is specially-coded regression is that the 
usual assumptions and diagnostics are appropriate in this context. You can plot 
residuals against estimated values, for example, to check for homogeneity of variance. 
»* d a [3 LI t * 
bs rary 
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Multigroup ANOVA 


When there are more groups, the coding of categories becomes more complex. For the 
effects model, there are one fewer coding variables than number of categories. For two 
categories, you need only one coding variable; for three categories, you need two 
coding variables: 


Category Code 
1 1 0 
2 0 1 


For the means model, the extension is straightforward: 


Category Code 

1 1 0 0 
2 0 1 0 
3 0 0 1 


For multigroup ANOVA, the models have the same form as for the two-group ANOVA 
above. The corresponding hypotheses for testing whether there are differences 
between means are: 


Ho: a¡=a,=03=0 (effects model) 
Ho: py = m = M3 (means model) 


You do not need to know how to produce coding variables to do ANOVA. SYSTAT 
does this for you automatically. All you need is a single variable that contains different 
values for each group. SYSTAT translates these values into different codes. It is 
important to remember, however, that regression and analysis of variance are not 
fundamentally different models. They are both instances of the general linear model. 


Factorial ANOVA 


It is possible to have more than one categorical variable in ANOVA. When this 
happens, you code each categorical variable exactly the same way as you do with 
multi- ANOVA. The coded design variables are then added as a full set of 
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ANOVA factors can interact. For example, a treatment may enhance bar pressing 
by male rats, yet suppress bar pressing by female rats. To test for this possibility, you 
can add (to your model) variables that are the product of the main effect variables 
already coded. This is similar to what you do when you construct polynomial models. 
For example, this is a model without an interaction: 


y = CONSTANT + treat + sex 


This is a model that contains interaction: 


y = CONSTANT + treat + sex + treat*sex 


If the hypothesis test of the coefficients for the TREA T*SEX term is significant, then 
you must qualify your conclusions by referring to the interaction. You might say, “Tt 
works one way for males and another for females.” 


Data Screening and Assumptions 


Most analyses have assumptions. If your data do not meet the necessary assumptions, 

then the resulting probabilities for the statistics may be suspect. Before an ANOVA, 

look for: 

m Violations of the equal variance assumption. Your groups should have the same 
dispersion or spread (their shapes do not differ markedly). 

m Symmetry. The mean of each group should fall roughly in the middle of the spread 
(the within-group distributions are not extremely skewed). 

m Independence of the group means and standard deviations (the size of the group 
means is not related to the size of their standard deviations). 


m Gross outliers (no values stand apart from the others in the batch). 


Graphical displays are useful for checking assumptions. For analysis of variance, try 
dit plots, box-and-whisker displays, or bar charts with standard error bars. 


Levene Test 


Analysis of variance assumes that the data within cells are independent and normally 
distributed with equal variances. This is the ANOVA equivalent of the regression 
assumptions for residuals. When the homogeneous variance part of the assumptions is 
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false, it is sometimes possible to adjust the degrees of freedom to produce an 
approximately distributed F-ratio. 

Levene (1960) proposed a test for unequal variances. You can use this test to 
determine whether you need an unequal variance F test. Simply fit your model in 
ANOVA and save residuals. Then transform the residuals into their absolute values. 
Merge these with your original grouping variable(s). Then redo your ANOVA on the 
absolute residuals. If it is significant, then you should consider using the separate 
variances test. 

Before doing all this work, you should do a box plot by groups to see whether the 
distributions differ. If you see few differences in the spread of the boxes, Levene’s test 
is unlikely to be significant. 


Pairwise Mean Comparisons 


The results in an ANOVA table serve only to indicate whether the means differ 
significantly or not. They do not indicate which mean differs from another. 

To report the pairs of means that differ significantly, you might think of computing 
a two-sample / test for each pair; however, do not do this. The probability associated 
with the two-sample / test assumes that only one test is performed. When several means 
are tested pairwise, the probability of finding one significant difference by chance 
alone increases rapidly with the number of pairs. If you use a 0.05 significance level to 
test that means 4 and B are equal and to test that means C and D are equal, the overall 
acceptance region is now 0.95 x 0.95, or 0.9025. Thus, the acceptance region for two 
independent comparisons carried out simultaneously is about 90%, and the critical 
region is 10% (instead of the desired 5%). For six pairs of means tested at the 0.05 
significance level, the probability of a difference falling in the critical region is not 0.05 
but 1 — (0.95)5 = 0.265. For 10 pairs, this probability increases to 0.40. The result of 
following such a strategy is to declare differences as significant when they are not. 

As an alternative to the situation described above, SYSTAT provides fifteen 
techniques to perform pairwise mean comparisons. You have to choose a proper test 
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based on the variance assumptions and the error rate to be controlled. SYSTAT offers 
the following tests divided into two parts based on variance assumptions: 


Equal Variance Unequal Variance 

Tukey Tamhane’s T2 
Bonferroni Games-Howell 
Fisher’s LSD Dunnett's T3 
Sidak 

Scheffé 

Tukey’s b 

Duncan 


Ryan-Einot-Gabriel-Welsch Q 
Hochberg’s GT2 

Gabriel 
Student-Newman-Keuls 
Dunnett 


The Student-Newman-Keuls procedure (S-N-K) and Duncan’s multiple range test 
control neither the individual nor the family-wise error rates. Duncan's test has been 
heavily criticized in the statistical literature; it gives many more statistically significant 
differences than is warranted and does not really protect the significance level. As a 
general rule, Fisher’s LSD is one of the more liberal procedures (more likely to declare 
means different), but it does not control the family-wise error rate. Tukey’s and 
Scheffe’s methods are conservative, with Scheffé’s method being more conservative 


than Tukey’s method. 


There is an abundance of literature covering multiple comparisons (see Miller, 1985); 

however, a few points are worth noting here: 

m Ifyou have a small number of groups, the Bonferroni pairwise procedure will often 
be more powerful (sensitive). For more groups, consider the Tukey method. Try all 
the methods in ANOVA (except Fisher’s LSD) and pick the best one. 

m Carrying out all possible pairwise comparisons is a waste of power. Think about a 
meaningful subset of comparisons and test this subset with Bonferroni levels. To 
do this, divide your critical level, say 0.05, by the number of comparisons you are 
making. You will almost always have more power than with any other pairwise 
multiple comparison procedure. 

= Some popular multiple comparison procedures do not maintain their claimed 
protection levels. Other stepwise multiple range tests, such as the Student- 
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Newman-Keuls and Duncan’s tests, have not been conclusively demonstrated to 
maintain overall protection levels for all possible distributions of means. 

= Some tests produce and test homogeneous subsets of group means instead of 
testing each pair of the group means. 

W Some tests come under unequal variance assumptions and use group variances 
instead of MSE to compare the group means. 


Linear and Quadratic Contrasts 


Contrasts are used to test relationships among means. A contrast is a linear 
combination of means p; with coefficients o: 


aih + opo +... og 0 


where a+ a) +... + a, = 0. In SYSTAT, hypotheses can be specified about contrasts 
and tests performed. Typically, the hypothesis has the form: 


Ho: eap, + aap +... + 04H 0 


The test statistic for a contrast is similar to that for a two-sample / test; the result of the 
contrast (a relation among means, such as mean A minus mean 8) is in the numerator 
of the test statistic, and an estimate of within-group variability (the pooled variance 
estimate or the error term from the ANOVA) is part of the denominator. 


You can select contrast coefficients to test: 
= Pairwise comparisons (test for a difference between two particular means) 


= A linear combination of means that are meaningful to the study at hand (compare 
two treatments versus a control mean) 


W Linear, quadratic, or the like increases (decreases) across a set of ordered means 
(that is, you might test a linear increase in sales by comparing people with no 
training, those with moderate training, and those with extensive training) 


Many experimental design texts place coefficients for linear and quadratic contrasts for 
three groups, four groups, and so on, in a table. SYSTAT allows you to type your 
contrasts or select a polynomial option. A polynomial contrast of order | is linear; of 
order 2, quadratic; of order 3, cubic; and so on. 
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Unbalanced Designs 


An unbalanced factorial design occurs when the numbers of cases in cells are unequal 
and not proportional across rows or columns. The following is an example of a 


2 x 2 design: 
Bl B2 

Al 1 5 
2 3 

4 

A2 6 2 
7 1 
9 5 
8 3 
4 


Unbalanced designs require a least-squares procedure like the General Linear Model 
because the usual maximum likelihood method of adding up the sum of squared 
deviations from cell means and the grand mean does not yield maximum likelihood 
estimates of effects. The General Linear Model adjusts for unbalanced designs when 
you get an ANOVA table to test hypotheses. 

However, the estimates of effects in the unbalanced design are no longer orthogonal 
(and thus statistically independent) across factors and their interactions. This means 
that the sum of squares associated with one factor depends on the sum of squares for 
another or its interaction. 

Analysts accustomed to using multiple regression have no problem with this 
situation because they assume that their independent variables in a model are 
correlated. Experimentalists, however, often have difficulty speaking of a main effect 
conditioned on another. Consequently, there is extensive literature on hypothesis 
testing methodology for unbalanced designs (for example, Speed and Hocking, 1976, 
and Speed, Hocking, and Hackney, 1978), and there is no consensus on how to test 
hypotheses with non-orthogonal designs. 

Some statisticians advise you to do a series of hierarchical tests beginning with 
interactions. If the highest-order interactions are insignificant, drop them from the 
model and recompute the analysis. Then, examine the lower-order interactions. If they 
are insignificant, recompute the model with main effects only. Some computer 
programs automate this process and print sum of squares and F tests according to the 
hierarchy (ordering of effects) you specify in the model. These are often called Type I 
sum of squares. 
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This procedure is analogous to stepwise regression in which hierarchical subsets of 
models are tested. This example assumes that you have specified the following model: 


Y = CONSTANT + a + b + c + a*b + atc + b*c + a*b*c 


The hierarchical approach tests the following models: 


Y = CONSTANT + a + b + C + a*b + atc + b*c + a*b*c 
Y = CONSTANT + a + b + C + atb + atc + b*c 

Y = CONSTANT + a + b + c + a*b + arc 

Y = CONSTANT + a + b + c + a*b 

Y = CONSTANT +a+b+c 

Y = CONSTANT + a + b 

Y = CONSTANT + a 


The problem with this approach, however, is that plausible subsets of effects are 
ignored if you examine only one hierarchy. The following model, which may be the 
best fit to the data, is never considered: 


Y = CONSTANT + a + b + a*b 


Furthermore, if you decide to examine all the other plausible subsets, you are really 
doing all possible subsets regression, and you should use Bonferroni confidence levels 
before rejecting a null hypothesis. The example above has 127 possible subset models 
(excluding ones without a CONSTANT). Interactive stepwise regression allows you to 
explore subset models under your control. 

If you have done an experiment and have decided that higher-order effects 
(interactions) are of enough theoretical importance to include in your model, you 
should condition every test on all other effects in the model you selected. This is the 
classical approach of Fisher and Yates. It amounts to using the default F values on the 
ANOVA output, which are the same as the Type III sum of squares. 

Probably the most important reason to stay with one model is that if you eliminate 
a series of effects that are not quite significant (for example, p — 0.06), you could end 
up with an incorrect subset model because of the dependencies among the sum of 
squares. In summary, if you want other sum of squares, compute them. You can supply 
the mean square error to customize sum of squares by using a hypothesis test in GLM, 
selecting MSE, and specifying the mean square error and degrees of freedom. 
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Repeated Measures 


In factorial ANOVA designs, each subject is measured once. For example, the 
assumption of independence would be violated if a subject is measured first as a 
control group member and later as a treatment group member. However, in a repeated 
measures design, the same variable is measured several times for each subject (case). 
A paired-comparison t test is the most simple form of a repeated measures design (for 
example, each subject has a before and after measure). 

Usually, it is not necessary for you to understand how SYSTAT carries out 
calculations; however, repeated measures is an exception. It is helpful to understand 
the quantities SYSTAT derives from your data. First, remember how to calculate a 
paired-comparison ¢ test by hand: 
= For each subject, compute the difference between the two measures. 
= Calculate the average of the differences. 
= Calculate the standard deviation of the differences. 

" 


Calculate the test statistic using this mean and standard deviation. 


SYSTAT derives similar values from your repeated measures and uses them in 
analysis-of-variance computations to test changes across the repeated measures 
(within subjects) as well as differences between groups of subjects (between subjects.) 
Tests of the within-subjects values are called polynomial tests of order 1, 2,..., up to k, 
where k is one less than the number of repeated measures. The first polynomial is used 
to test linear changes (for example, do the repeated responses increase (or decrease) 
around a line with a significant slope?). The second polynomial tests if the responses 
fall along a quadratic curve, and so on. 

For each case, SYSTAT uses orthogonal contrast coefficients to derive one 
number for each polynomial. For the coefficients of the linear polynomial, SYSTAT 
uses (-1, 0, 1) when there are three measures; (—3, -1, 1, 3) when there are four 
measures; and so on. When there are three repeated measures, SYSTAT multiplies the 
first by —1, the second by 0, and the third by 1, and sums these products (this sum is 
then multiplied by a constant to make the sum of squares of the coefficients equal to 
1). Notice that when the responses are the same, the result of the polynomial contrast 
is 0; when the responses fall closely along a line with a steep slope, the polynomial 
differs markedly from 0. 

For the coefficients of the quadratic polynomial, SYSTAT uses (1, 2, 1) when 
there are three measures; (1, —], -1, 1) when there are four measures; and so on. The 
cubic and higher-order polynomials are computed in a similar way. 
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Let us continue the discussion for a design with three repeated measures. Assume 
that you record body weight once a month for three months for rats grouped by diet. 
(Diet A includes a heavy concentration of alcohol and Diet B consists of normal lab 
chow.) For each rat, SYSTAT computes a linear component and a quadratic 
component. SYSTAT also sums weights to derive a total response. These derived 
values are used to compute two analysis of variance tables: 


m The rotal response is used to test between-group differences; that is, the total is 
used as the dependent variable in the usual factorial ANOVA computations. In the 
example, this test compares total weight for Diet A against that for Diet B. This is 
analogous to a two-sample ¢ test using total weight as the dependent variable. 


m The linear and quadratic components are used to test changes across the repeated 
measures (within subjects) and also to test the interaction of the within factor with 
the grouping factor. If the test for the linear component is significant, you can 
report a significant linear increase in weight over the three months. If the test for 
the quadratic component is also significant (but much less so than the linear 
component), you might report that growth is predominantly linear, but there is a 
significant curve in the upward trend. 


m A significant interaction between Diet (the between-group factor) and the linear 
component across time might indicate that the slopes for Diet A and Diet B differ. 
This test may be the most important one for the experiment. 


Assumptions in Repeated Measures 


SYSTAT computes both univariate and multivariate statistics. Like all standard 
ANOVA procedures, the univariate repeated measures approach requires that the 
distributions within cells be normal. The univariate repeated measures approach also 
requires that the covariances between all possible pairs of repeated measures be equal. 
(Actually, the requirement is slightly less restrictive, but this difference is of little 
practical importance.) Of course, the usual ANOVA requirement that all variances 
within cells are equal still applies; thus, the covariance matrix of the measures should 
have a constant diagonal and equal elements off the diagonal. This assumption is called 
compound symmetry. 

The multivariate analysis does not require compound symmetry. It requires that the 
covariance matrices within groups (there is only one group in this example) be 
equivalent and that they be based on multivariate normal distributions. If the classical 
assumptions hold, then you should generally ignore the multivariate tests at the bottom 
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of the output and stay with the classical univariate ANOVA table because the 
multivariate tests will be generally less powerful. 

There is a middle approach. The Greenhouse-Geisser and Huynh-Feldt statistics are 
used to adjust the probability for the classical univariate tests when compound 
symmetry fails. (Huynh-Feldt is a more recent adjustment to the conservative 
Greenhouse-Geiser statistic.) If the Huynh-Feldt p-values are substantially different 
from those under the column directly to the right of the F-ratio, then you should be 
aware that compound symmetry has failed. In this case, compare the adjusted p-values 
under Huynh-Feldt to those for the multivariate tests. 

If all else fails, single degree-of-freedom polynomial tests can always be trusted. If 
there are several to examine, however, remember that you may want to use Bonferroni 
adjustments to the probabilities; that is, divide the normal value (for example, 0.05) by 
the number of polynomial tests you want to examine. You need to make a Bonferroni 
adjustment only if you are unable to use the summary univariate or multivariate tests 
to protect the overall level; otherwise, you can examine the polynomials without 
penalty if the overall test is significant. 

See Timm (2002) for a discussion on repeated measures. 


Issues in Repeated Measures Analysis 


Repeated measures designs can be generated in SYSTAT with a single procedure. You 
need not worry about weighting cases in unbalanced designs or selecting error terms. 
The program does this automatically; however, you should keep the following in mind: 


m The sum of squares for the univariate F tests are pooled across subjects within 
groups and their interactions with trials. This means that the traditional analysis 
method has highly restrictive assumptions. You must assume that the variances 
within cells are homogeneous and that the covariances across all pairs of cells are 
equivalent (compound symmetry). There are some mathematical exceptions to this 
requirement, but they rarely occur in practice. Furthermore, the compound 
symmetry assumption rarely holds for real data. 


= Compound symmetry is not required for the validity of the single degree-of- 
freedom polynomial contrasts. These polynomials partition sum of squares into 
orthogonal components. You should routinely examine the magnitude of these sum 
of squares relative to the hypothesis sum of squares for the corresponding 
univariate repeated measures F test when your trials are ordered on a scale. 


m Think of the repeated measures output as an expanded traditional ANOVA table. 
The effects are printed in the same order as they appear in Winer, Brown and 
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Michels (1991) and other texts, but they include the single degree-of-freedom and 
multivariate tests to protect you from false conclusions. If you are satisfied that 
both are in agreement, you can delete the additional lines in the output file. 


= You can test any hypothesis after you have estimated a repeated measures design 
and examined the output. For example, you can use polynomial contrasts to test 
single degree-of-freedom components in an unevenly spaced design. You can also 
use difference contrasts to do post hoc tests on adjacent trials. 


SYSTAT’s Sum of Squares 


SYSTAT provides several types of sum of squares for testing hypotheses. The 
following names for these sum of squares are not statistical terms, but they were 
popularized originally by SAS GLM. 


Type I. Type I sum of squares are adjusted for those terms which appear in the model 
after the term in question; some books refer to the method of obtaining this type of sum 
of squares as the hierarchical decomposition or the sequential sum of squares method 
(Milliken and Johnson, 2004). Type I sum of squares are computed from the difference 
between the residual sum of squares of two different models. The particular models 
needed for the computation depend on the order of the variables in the MODEL 
statement. 


For example, if the model is: 
MODEL y = CONSTANT + a + b + a*b 


then the sum of squares for A*B is produced from the difference between SSE (sum of 
squared error) in the two following models: 


MODEL y 
MODEL y 


CONSTANT + a + b 
CONSTANT + a + b + a*b 


Similarly, the Type I sum of squares for B in this model is computed from the 
difference in SSE between the following models: 


MODEL y = CONSTANT + a 
MODEL y = CONSTANT + a + b 
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Finally, the Type I sum of squares for 4 is computed from the difference in residual 
sum of squares for the following: 

MODEL y = CONSTANT 

MODEL y = CONSTANT + a 


In summary, to compute sum of squares, move from right to left and construct models 
which differ by the right-most term only. Type I sum of squares are commonly used for: 


= A balanced ANOVA model where effects are specified in a hierarchical manner, 
viz., the main effect, first order interaction, second order interactions, and so on. 


m A polynomial regression model in which terms in the model are ordered as per their 
degree. 

m A purely nested model in which the effect is specified in the proper order. 

Type II. Type II sum of squares is computed similarly to Type I except that main effects 

and interactions determine the ordering of differences instead of the MODEL statement 

order. For the above model, Type II sum of squares for the interaction is computed from 

the difference in residual sum of squares for the following models: 


MODEL y = CONSTANT + a + b 
MODEL y = CONSTANT + a + b + a*b 
For the B effect, difference the following models: 
MODEL y = CONSTANT + a + b 
MODEL y = CONSTANT + a 
For the A effect, difference the following (this is not the same as for Type I): 
MODEL y = CONSTANT + a + b 
MODEL y = CONSTANT + b 


In summary, include interactions of the same order as well as all lower order 
interactions and main effects when differencing to get an interaction. When getting 
sum of squares for a main effect, difference against all other main effects and 
interactions involved with these main effects.The Type II sum of squares method is 


commonly used for: 

m ANOVA model with unbalanced cell sizes (unbalanced ANOVA) 
m ANOVA model that has main effects only 

= Any regression model 

m Nested design 
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Type HI. Type III sum of squares are the default for ANOVA and are much simpler to 
understand. Simply difference from the full model, leaving out only the term in 
question. For example, the Type III sum of squares for A is taken from the following 
two models: 


MODEL y = CONSTANT + b + a*b 
MODEL y = CONSTANT + a + b + a*b 


The Type III sum of squares method is commonly used for: 
m Any models in Type I and Type II 

m Any balanced or unbalanced ANOVA 

m Any ANOVA models with no missing cells 


Type IV. Type IV sum of squares are designed for the missing cells designs and are not 
easily presented in the above terminology. They are produced by balancing over the 
means of nonmissing cells not included in the current hypothesis. SYSTAT has options 
to choose from three types of sum of squares, i.e. Type I, Type II, and Type III; you can 
choose one of these sum of squares for the analysis. 


By default SYSTAT produces Type III sum of squares. The user should take care in 
choosing the appropriate choice for the sum of squares. There is often a strong 
temptation to choose the most significant sum of squares without understanding the 
hypothesis being tested. 


Finally, Type IV is produced by the careful use of SPECIFY in testing means models. 
The advantage of this approach is that the user is always aware that sums of squares 
depend on explicit mathematical models rather than additions and subtractions of 
dimensionless quantities. 
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2 
Linear Models I: Linear Regression 


Leland Wilkinson and Mark Coward 
(revised by Soumyajit Ghosh and S.R.Kulkarni) 


The model for simple linear regression is: 


y= +Bxte 


where y is the dependent variable, x is the independent variable, and the f)'s are the 
regression parameters (the intercept and the slope of the line of best fit). The model 
for multiple linear regression is: 


y=Bot+Bix +B,x,+...+B,x, +E 


The Linear Regression feature offers three methods for fitting a multiple linear 
regression model; Least Squares Regression, Ridge Regression, and Bayesian 
Regression. Least Squares Regression estimates and tests simple and multiple linear 
regression models. The ability to do stepwise regression is available in three ways: 
use the default values, specify your own selection criteria, or at each step, 
interactively select a variable to add or remove from the model. SYSTAT offers three 
tests for checking normality: Kolmogorov-Smirnov Lilliefor's test, Shapiro-Wilk test, 
and Anderson-Darling test, if opted. For each model you fit in Least Squares 
Regression, SYSTAT reports RÀ, adjusted R?, the standard error of the estimate, and 
an ANOVA table for assessing the fit of the model. AIC, AIC (C orrected) and 
Schwarz's BIC values are also provided for each fitted model. For more information 
on AIC and Schwarz's (1978) BIC refer to Chapter 1: Linear Models, “Variable 
Selection“ on page 15 in Statistics IT. For each variable in the model, the output 
includes the estimate of the regression coefficient, the standard error of the 
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coefficient, the standardized coefficient, tolerance, variance inflation factor (V/F), and 
at statistic for measuring the usefulness of the variable in the model. A plot of residuals 
against the predicted values is provided. Also, in the case of single-predictor 
(independent variable), a plot of the fitted regression line with confidence limits for a 
single mean response and prediction limits for new observations is provided, and only 
fitted model in case of two predictors. 

When the predictor variables are correlated, i.e. when multicollinearity exists, the 
least-squares estimates of regression coefficients tend to have a large sampling 
variability. In such a situation, ridge regression offers a method to obtain better 
estimates of regression coefficients. Two types of ridge coefficients: standardized 
coefficients and unstandardized coefficients are computed. A plot of the ridge factor 
against the ridge coefficients is also available. The technique of Partial Least Squares 
(PLS) regression can also be a remedy for multicollinearity. PLS reduces the 
dimensionality of the regression problem by using linear combinations of the 
predictors, in the process of which multicollinearity may be reduced or removed. For 
details of PLS, see “Partial Least Squares Regression" on page 357 in Statistics III. 

Bayesian regression provides another paradigm for fitting a multiple linear 
regression model. The prior distribution for the regression parameters used in this 
feature is a (multivariate) Normal-Gamma distribution or a diffuse. Bayes estimates 
and credible intervals for the regression coefficients are computed. Also, the 
parameters of the posterior distribution are provided along with plots of prior and 
posterior densities of the regression coefficients. 

Resampling procedures are available only with Least Squares Regression. SY STAT 
gives a summarization based on resampling for Linear Regression. You can get 
resampling-based estimates of the regression coefficients along with their bias and 
standard error. Under bootstrap, you will also get confidence intervals of coefficients 
using two popular methods, viz., Percentile method and Bias corrected and accelerated 
method. 


11-41 


Linear Models I; Linear Regression 


Linear Regression in SYSTAT 


Least Squares Regression Dialog Box 


To open Least Squares Regression dialog box, from the menus choose: 


Analyze 
Regression 
Linear 
Least Squares... 


Regression: Linear: Least 5quares 
Model | Estimation|| Options | Predict Resa 


Available > variable(s): _ Dependent: 


SALBEG 
SEX 
TIME 
AGE EJ 


Independent(s): 


Wi 5b | <Requied 


[Save  ¡Resduale 


The following options can be specified: 


Include constant. Includes the constant in the regression equation. Deselect this option 
to remove the constant. You almost never want to remove the constant, and you should 


be familiar with no-constant regression terminology before considering it. 
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Cases. If your data are in the form of a correlation matrix, enter the number of cases 
used to compute the correlation matrix. 


Save. You can save residuals and other data to a new data file. The following 
alternatives are available: 


Adjusted. Saves the adjusted estimates of the regression coefficients. 


Adjusted/data. Saves the adjusted estimates plus all the variables in the working 
data file. 


Coefficients. Saves the estimates of the regression coefficients. 
Model. Saves statistics given in Residuals and the variables used in the model. 
Partial. Saves partial residuals. Suppose your model is: 


Y=CONSTANT + X1 + X2 + X3 


The saved file contains: 


YPARTIAL (1) : Residual of Y = CONSTANT + X2 + X3 
XPARTIAL (1) : Residual of X1 = CONSTANT + X2 + X3 
YPARTIAL (2) : Residual of Y = CONSTANT + X1 + X3 


XPARTIAL (2) : Residual of X2 = CONSTANT + X1 + X3 
YPARTIAL (3) : Residual of Y = CONSTANT + X1 + X2 
XPARTIAL (3) : Residual of X3 = CONSTANT + X1 + X2 


Partial/data. Saves partial residuals plus all the variables in the working data file, 
including any transformed data values, 


Residuals. Saves predicted values, residuals, Studentized residuals, leverage for 
each observation, Cook’s distance measure, standard error of predicted values, and 
confidence and prediction intervals for the response variable. 


Residuals/data. Saves the residual statistics given by Residuals plus all the 
variables in the working data file, including any transformed data values. 


Estimation 


To specify the Estimation option, click the Estimation tab in the Least Squares 
Regression dialog box. 
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Regression: Linear: Least Squares 


e a E ua 
Model | Estimation | Iptions ef edict iE € 


Linear Models I: Linear Regression 


Tolerance: |1e012 | Stepwise options: 


Confidence: | 0.95 
Estimation 
O Complete 
© Stepwise 


Mixture model 


You can specify confidence and a tolerance level, select complete or stepwise entry, 
and specify entry and removal criteria. 


Tolerance. Prevents the entry of a variable that is highly correlated with the 
independent variables already included in the model. Enter a value between 0 and 1. 
Typical values are 0.01 or 0.001. The higher the value (closer to 1), the lower the 
correlation required to exclude a variable. 


Confidence. Specifies regression coefficients and response variable at the desired level 
of confidence. The default value is 0.95. 


Estimation. Controls the method used to enter and remove variables from the equation. 


m Complete. All independent variables are entered in a single step. 
m Mixture model. Constrains the independent variables to sum to a constant. 


m Stepwise. Variables are entered or removed from the model one at a time. 
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Stepwise options. The following alternatives are available for stepwise entry and 
removal: 


Ww Backward. Begins with all candidate variables in the model. At each step, 
SYSTAT removes the variable with the largest Remove value. 


W Forward. Begins with no variables in the model. At each step, SYSTAT adds the 
variable with the smallest Enter value. 


W Automatic. For Backward, at each step SYSTAT automatically removes a variable 
from your model. For Forward, SYSTAT automatically adds a variable to the model 
at each step. 


W Interactive. At each step in the model building, you select the variable to enter or 
remove from the model. 
You can also control the criteria used to enter and remove variables from the model: 


W Probability. Specify probabilities to enter and to remove variable from the model. 
A variable is entered into the model if its alpha value is less than the specified Enter 
value and is removed from the model if its alpha value is greater than the specified 
Remove value. Specify values between 0 and 1. 


m F-ratio. Specify F-to-enter and F-to-remove limits. Variables with F-ratio greater 
than the specified value are entered into the model if Tolerance permits and 
variables with F-ratio less than the specified value are removed from the model. 


m MaxStep. Maximum number of steps. 
m Force. Force the first n variables listed in your model to remain in the equation. 
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Options 


To specify the options, click the Options tab in the Least Squares Regression dialog 
box. 


Regression: Linear: Least Squares 


| Model | Estimation] Options | Predict! Re 


Normality tests 
Kolmogorov-Smimnov Shapiro-Wilk 


Anderson-Darling 


Normality tests. You can use the following tests to check the normality of residuals: 


m Kolmogorov-Smirnov. It’s a nonparametric test used for large samples. It is 
applied to continuous distributions and gives greater importance to the 
observations in the center than those at the tails. 

m Shapiro-Wilk. The test provides Shapiro-Wilk test statistic and p-value for the 
residuals: the smaller the p-value, the worse is the fit. 

m Anderson-Darling. Anderson-Darling test is a standard goodness of fit test. It gives 


greater importance to the observations in the tails than those at the center. 
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Predict 


To predict the new values, click the Predict tab in the Least Squares regression dialog 
box. 


Regression: Linear: Least Squares 


Predict for new observation(s]: 


Save | Prediction Y 
Confidence: 


The following options can be specified: 


Prediction for new observation(s). Predicts the dependent variable value for given 
values of the predictors. 


Confidence. Displays the confidence and prediction intervals at the desired level of 
confidence. The default value is 0.95, 


Save. You can save predicted values and new data on to a new data file. The following 
alternatives are available: 


W Prediction. Saves the predicted values, standard errors of predicted values, lower 
and upper confidence limits of predicted values. 
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m Prediction/New Data. Saves the statistics given by Prediction plus variables in the 
model in new data file. 


Resampling 


Click the Resampling tab to specify different resampling options. 


Regression: Linear: Least Squares 


| Model | Estimation|| Options!! Predict| Resampling 


Perform resampling 


Method: Bootstrap 


Number of samples: 
Sample size: 


Random seed: 


Confidence: 


Perform resampling. Generates samples of cases and uses data thereof to carry out the 
same analysis on each sample. 

Method. Three sampling methods are available: 

m Bootstrap. Generates bootstrap samples. This is the default method. 

m Without replacement. Generates subsamples without replacement. 


m Jackknife. Generates jackknife samples. 
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Number of samples. Specify the number of samples to be generated. These samples are 
analyzed using the chosen method of sampling. The default is 1. 


Sample size. Specify the size of each sample to be generated while resampling. The 
default sample size is the number of cases in the data file in use. 


Random seed. Specify a random seed to be used while resampling. The default random 
seed is generated by the system. 


Confidence. Specify a confidence level for bootstrap-based confidence interval. Enter 
any value between 0 and 1. The default is 0.95. 


Ridge Regression 


Ridge regression is one of the several methods that have been proposed as a remedy 
for multicollinearity problems. It is useful when small values are desired for the least- 
squares regression coefficients in situations like when the sum of squares of the 
regression coefficients is bounded above. A clue to the need for ridge regression is 
obtained when the smallest eigenvalue of the X'X matrix is much less than | and the 
variance inflation factors (VIF) are large. 

A ridge estimator of regression coefficients is obtained by modifying the method 
of least-squares (this is done by introducing a constant 'lambda' in the normal 
equations) to allow shrunken and biased estimators of regression coefficients. 
SYSTAT computes two estimates of lambda: HKB estimate proposed by Hoerl, 
Kennard, and Baldwin (1975), and LW estimate proposed by Lawless and Wang 
(1976). Though the ridge estimator is a biased estimator of regression coefficients, its 
mean square error is smaller than that of the least-squares estimator. 
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Ridge Regression Dialog Box 


To open the Ridge Regression dialog box, from the main menus choose: 


Analyze 
Regression 
Linear 
Ridge... 


ssion:Linear:Ridge 


Dependent. The variable to be predicted. The dependent variable should be 
quantitative in nature. 

Independent(s). Select one or more variables. Normally, there exists high collinearity 
between the variables. 

Lambda. You can specify the values of lambda to get HKB and LW estimates of 
optimal values of lambda. 
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You can specify individual lambda values or a range of lambda values to get the HKB 
and LW estimates of optimal values of lambda. Lambda is a real variable. 


m Range of values. Specify a range of lambda values. The following options are 
provided for specifying the range of lambda values: 


W Minimum. Enter the minimum value or the start value of lambda. 
m Maximum. Enter the maximum value or the end value of lambda. 
W Increment. Specify the difference between consecutive values. 

m Individual values. Specify desired set of lambda values. 


Save coefficient(s). Saves standardized ridge coefficients corresponding to each of the 
lambda values to a file. 


Bayesian Regression 


In the Bayesian approach, estimates of the regression parameters in a multiple linear 
regression model are obtained by incorporating prior information in the form of a prior 
distribution of the parameters. In classical Bayesian analysis, a widely used choice of 
the prior distribution is the (multivariate) Normal-Gamma distribution when the error 
component has a normal distribution. An advantage of this choice is that it is a 
conjugate prior, resulting in the form of the posterior distribution of the regression 
parameters to be the same as that of the prior distribution. 

The Bayesian approach has one more advantage in that it produces a direct 
probability statement about a parameter in the form of credibility intervals. For more 
information on Bayesian regression, see Zellner (1971), Box and Tiao (1973) and Press 
(1989). 
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To obtain Bayesian Regression dialog box, from the menus choose: 


Analyze 
Regression 
Linear 
Bayesian... 


Regression:Linear:Bayesian 
Available variables; — Dependent: 
| WEIGHT | «Required» 

| <- Remove 


Independent{s}: 


«Required: 


Add => 
| EDLEVEL | <= Remove | 
| WORK 
O Diffuse prior 
© Notmal-gamma prior 

Normal prior parameters 
Mean vector "m 


@Fromkeyboard | - 
O From file 
Covariance matrix 


(O) From keyboard pa 
O From file | 


Credibility: fos — ] 


[Save | Coefficients 


oK 


Dependent. Select the variable you want to predict. The dependent variable should be 


continuous and numeric. 
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Independent. Select one or more independent variables. 


Include constant. Includes the constant in the model (by default). Uncheck the box if 
you do not want to include the constant term in your regression equation. 


Diffuse prior. Uses diffuse priors for estimation. 
Normal-Gamma prior. Specify the Normal-Gamma conjugate priors for Bayesian 
estimation of regression coefficients. 


= Normal prior parameters. Specify the parameters of the prior distribution of 
regression parameters. 

W Mean vector. Enter the mean vector of the multivariate normal prior distribution of 
regression parameters either through the keyboard or using a file. 


W Covariance matrix. Enter the covariance matrix of the multivariate normal prior 
distribution of regression parameters either through the keyboard or using a file. 


= Gamma prior parameters. Enter the values of the scale and shape parameters of 
the gamma prior distribution for the inverse of the variance. The selection of 
gamma prior is optional. If one doesn't specify any gamma priors, only the 
regression coefficients of the posterior distribution are obtained. 


Credibility. Enter the credibility coefficient (Bayesian analog of the confidence 
coefficient) to get the desired percentage credible interval. The default is 0.95. 


Save. The following alternatives are available: 


= Coefficients. Saves the estimates of the Bayesian regression coefficients to a 
specified file. 


Residuals/data. Saves all the predicted values, residuals and the original data. 


m Conditional covariance matrix. Saves the conditional covariance matrix of 
Bayesian regression coefficients given sigma. 


= Marginal covariance matrix. Saves the marginal covariance matrix of Bayesian 


regression coefficients. 
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Using Commands 


For least squares regression 


First, specify your data with USE filename. Continue with: 


REGRESS 
MODEL var-CONSTANT + varl + var2 + .. / N-n 
SAVE filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 
WORK filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 
ESTIMATE /MIX TOL-n NTEST - KS, SW, AD Quick NoQuick 
SAVE filename / PREDICT NEWDATA 
WORK filename / PREDICT NEWDATA 
PREDICT filename /Confi-n Quick NoQuick 


(use START instead of ESTIMATE for stepwise model building) 


SAVE filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 

START / FORWARD BACKWARD TOL-n ENTER-p REMOVE-p, 
FENTER-n FREMOVE-n FORCE-n 

STEP / AUTO ENTER-p REMOVE-p FENTER-n FREMOVE-n 

STOP/ Quick NoQuick 


For getting the summarized resampling output, the following command should be 
given before the ESTIMATE command. 


SAMPLE BOOT(m,n) or SIMPLE(m,n) or JACK / CONFI - c 


For ridge regression 


Select a data file using USE filename and continue with: 


RIDGEREG 
MODEL var = CONSTANT + varl + var2 +..+ varn 
SAVE filename 
WORK filename 
ESTIMATE / LMIN=a LMAX=b LSTEP=c or LAMBDA=11, 12,.., 1k 
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For Bayesian regression 


BAYESIAN 
MODEL var = CONSTANT + varl + var2 +..+ varn 
SAVE filename/COEFFICIENTS or RESIDUALS, DATA or CONDITIONAL 
or MARGINAL 
WORK filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 
ESTIMATE / MEAN = [b] or 'filenamel' VAR = [v] or 'filename2' 
SCALE=a SHAPE=c CREDIBILITY=d 


Usage Considerations 


Types of data. REGRESS uses the usual cases-by-variables data file or a covariance, 
correlation, or sum of squares and cross products matrix. Using matrix input requires 
specification of the sample size, which generated the matrix. RIDGEREG and 
BAYESIAN use rectangular data only. 


Print options. For REGRESS, using PLENGTH MEDIUM, the output includes 
eigenvalues of X'X, condition indices, and variance proportions. PLENGTH LONG adds 
the correlation matrix of the regression coefficients to this output. For RIDGEREG and 
BAYESIAN regression, the output is standard for all PLENGTH options. 


Quick Graphs. REGRESS plots the residuals against the predicted values. Also plots 
confidence limits for a single mean response and prediction limits for new observations 
in the single-predictor case for both original and new observations. And in case of two 
predictors it plots a fitted model. RIDGEREG plots a graph between the ridge factor and 
the ridge coefficients. BAYESIAN produces plots of the prior and the posterior densities 
of each regression coefficient and of the variance. 


Saving files. REGRESS saves the results of the analysis (predicted values, residuals, 
confidence intervals, prediction intervals and diagnostics that identify unusual cases). 
RIDGEREG saves the ridge coefficients and BAYESIAN saves the estimates of the 
regression coefficients, residuals marginal and conditional covariance matrix. 


BY groups. REGRESS, RIDGEREG, and BAYESIAN analyze data by groups. 


Case frequencies. REGRESS, RIDGEREG, and BAYESIAN use the FREQ variable to 
duplicate cases. This inflates the degrees of freedom to be the sum of frequencies. 


Case weights. REGRESS, RIDGEREG and BAYESIAN weight cases using the WEIGHT 
variable for rectangular data. You can perform cross-validation if the weight variable 
is binary and coded 0 or 1. SYSTAT computes predicted values for cases with zero 
weight even though they are not used to estimate the regression parameters. 
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Examples 


Example 1 
Simple Linear Regression 


In this example, we explore the relation between gross domestic product per capita 
(GDP. CAP) and spending on the military (MIL) for 57 countries that report this 
information to the United Nations—we want to determine whether a measure of the 
financial well being of a country is useful for predicting its military expenditures. Our 
model is: 


mil=P, +B,gdp_cap+e 


Initially, we plot the dependent variable against the independent variable. Such a plot 
may reveal outlying cases or suggest a transformation before applying linear 
regression. 


The input is: 


USE OURWORLD 
IF COUNTRY$='IRAQ' or COUNTRY$='LIBYA' THEN LET NAMES$-COUNTRY$ 
PLOT MIL*GDP CAP / SMOOTH-LOWESS TENSION 20.500, 
T; YLABEL-'Military Spending', 
SYMBOL=4 SIZE- 1.500 LABEL-NAMES, 
CSIZE-2.000 
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The output is: 
800 
700 
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o 
£ 
"o 500 
Hd 
Y) 400 
j^ 
ER 
z 
200 
100 
0 5000 10000 15000 20000 
GDP CAP 


To obtain the scatterplot, we created a new variable, NAMES, that had missing values 
for all countries except Libya and Iraq. We then used the new variable to label plot 
points. 

Iraq and Libya stand apart from the other countries —they spend considerably more 
for the military than countries with similar GDP_CAP values. The smoother indicates 
that the relationship between the two variables is fairly linear. Distressing, however, is 
the fact that many points clump in the lower left corner. Many data analysts would 
want to study the data after log-transforming both variables. We do this in another 
example, but now we estimate the coefficients for the data as recorded. 


The input is: 


REGRESS 
USE OURWORLD 
ID COUNTRY$ 
MODEL MIL - CONSTANT « GDP CAP 
ESTIMATE 


157 


The output is: 


Linear Models I: Linear Regression 


1 case(s) are deleted due to missing data. 


Eigenvalues of Unit Scaled X'X 


1.681 0.319 
Condition Indices 
1.000 2.294 
Variance Proportions 
i 1 2 


0.840 
0.840 


CONSTANT 
GDP CAP | 


0.160 
0.160 


Dependent Variable 
N 


Multiple R 

Squared Multiple R 
Adjusted Squared Multiple R 
Standard Error of Estimate 


i 
1 
; 
! 
i 
i 


0.407 
136.154 


Regression Coefficients B = (X'X) X'Y 


i Std. 
Effect ! Coefficient Standard Error Coefficient Tolerance t 
--------- An ÓN 
CONSTANT ; 41.857 24.838 0.000 ` 1.685 
GDP CAP | 0.019 0.003 0.646 1.000 6.220 
Regression Coefficients B = (x'X) -ly'y (contd...) 
Effect H lue 
CONSTANT | 0.098 
GDP CAP | 0.000 
Confidence Interval for Regression Coefficients 

95.0$ Confidence Interval 

Effect Coefficient Lower Upper VIF 
CONSTANT 41.857 -7.940 91.654 " 
GDP CAP 0.019 0.013 0.025 1.000 
Analysis of Variance 
Source 1 SS df Mean Squares F-ratio p-value 
Regression i 717100.891 1 717100.891 38.683 0.000 
Residual i 1.001E«006 54 18537.876 


*** WARNING *** : 


Case Iraq 


is an Outlier (Studentized Residual : 6.956) 
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Case Libya is an Outlier (Studentized Residual : 4.348) 


Durbin-Watson D Statistic i 2.046 
First Order Autocorrelation | -0.032 


Information Criteria 


AIC } 713.229 
AIC (Corrected) | 713.690 
Schwarz's BIC | 719.305 

Plot of Residuals vs Predicted Values 
1000 


RESIDUAL 


~ ESTIMATE 
— LCL 
— UCL 
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SYSTAT reports that data are missing for one case. In the next line, it reports that 56 
cases are used (N = 56). In the regression calculations, SYSTAT uses only the cases 
that have complete data for the variables in the model. However, when only the 
dependent variable is missing, SYSTAT computes a predicted value, its standard error, 
and a leverage diagnostic for the case. In this sample, Afghanistan did not report 
military spending. 

When there is only one independent variable, Multiple R (0.646) is the simple 
correlation between MIL and GDP. CAP. Squared multiple R (0.417) is the square of 
this value, and it is the proportion of the total variation in the military expenditures 
accounted for by GDP. CAP (GDP. CAP explains 41.7% of the variability of MIL). 
Use Sum-of-Squares (SS) in the analysis of variance table to compute it: 


717100.891 / (717100.891 + 1001045.288) 


Adjusted squared multiple R is of interest for models with more than one independent 
variable. Standard error of estimate (136.154) is the square root of the residual mean 
square (18537.876) in the ANOVA table. 


The estimates of the regression coefficients are 41.857 and 0.019, so the equation is: 
mil = 41.857 + 0.019 * gdp cap 


The Standard Error of the estimated coefficients are in the next column and the 
standardized coefficients (Std Coefficient) follow. The latter are called beta weights by 
some social scientists. Tolerance and Variance inflation factor (VIF) are not relevant 
when there is only one predictor. 

Next are £ statistics (1) —the first (1.685) tests the significance of the difference of 
the constant from 0 and the second (6.220) tests the significance of the slope, which is 
equivalent to testing the significance of the correlation between military spending and 
GDP CAP. 

F-ratio in the analysis of variance table is used to test the hypothesis that the slope 
is 0 (or, for multiple regression, that all slopes are 0). The F-ratio is large when the 
independent variable(s) helps to explain the variation in the dependent variable. Here, 
there is a significant linear relation between military spending and GDP. CAP. Thus, 
we reject the hypothesis that the slope of the regression line is zero (F-ratio — 38.683, 
p-value « 0.0005). 

It appears from the results above that GDP CAP is useful for predicting spending 
on the military—that is, countries that are financially sound tend to spend more on the 
military than poorer nations. These numbers, however, do not provide the complete 
picture. Notice that SYSTAT warns us that the two countries (Iraq and Libya) with 
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unusual values could be distorting the results. We recommend that you consider 
transforming the data and that you save the residuals and other diagnostic statistics. 


Example 2 
Transformations 


The data in the scatterplot in the simple linear regression example are not well suited 
for linear regression, as the heavy concentration of points in the lower left corner of the 
graph shows. Here are the same data plotted in log units. 


The input is: 


REGRESS 
USE OURWORLD 
PLOT MIL*GDP_CAP / SMOOTH=LOWESS TENSION =0.500, 


XLABEL='GDP per capita', 

XLOG=10 YLABEL='Military Spending' YLOG=10, 
SYMBOL=4,2,3,SIZE= 1.250 LABEL=COUNTRY$, 
CSIZE=1.450 


The output is: 


8 


Military Spending 


GDP per capita 
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Except possibly for Iraq and Libya, the configuration of these points is better for linear 
modeling than that for the untransformed data. 


We now transform both the y and x variables and refit the model, the input is: 


REGRESS 
USE OURWORLD 
LET LOG MIL = L10 (MIL) 
LET LOG GDP - L10(GDP CAP) 
MODEL LOG MIL - 
ESTIMATE 


The output is: 


1 case(s) are deleted due to missing data. 


Eigenvalues of Unit Scaled X'X 


1.000 11.005 


Variance Proportions 


1 2 
CONSTANT ; 0.008 0.992 
LOG GDP 0.008 0.992 
Dependent Variable f TRD 
N 15 
Multiple R 1 0.857 
Squared Multiple R } 0.734 
Adjusted Squared Multiple R | 0.729 
Standard Error of Estimate | 0.346 


Regression Coefficients B = (X'X)' X'Y 


E i ffi Standard Error 
e: q 

CONSTANT |; -1. 0.257 
LOG GDP } 0.909 0.075 


CONSTANT + LOG_GDP 


Std. 
Coefficient Tolerance 
0.000 . 
0.857 1.000 


Regression Coefficients B = (X'X)'X'Y (contd...) 


Effect H p-value 
mates AA ieri ced 
CONSTANT |; 0.000 
LOG_GDP | 0.000 


Confidence Interval for Regression Coefficients 


| Coefficient 


Effect 


CONSTANT 
LOG GDP 


35.0% Confidence Interval 


Upper 
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Correlation Matrix of Regression Coefficients 


CONSTANT LOG GDP 


CONSTANT 
LOG GDP 


Mean Squares F-ratio p-value 


Regression 17.868 148.876 0.000 


Residual | 6.481 54 0:120 
ARA WARNING *** : 
Case 22 is an Outlier (Studentized Residual : 4.004) 


Durbin-Watson D Statistic i 
First Order Autocorrelation } 


Information Criteria 
AIC | 44.160 


AIC (Corrected) | 44.621 
Schwarz's BIC 1 50.236 


Plot of Residuals vs Predicted Values 


RESIDUAL 


The Squared multiple R for the variables in log units is 0.734 (versus 0.417 for the 
untransformed values). That is, we have gone from explaining 41.796 of the variability 
of military spending to 73.4% by using the log transformations. The F-ratio is now 
148.876—it was 38.683. Notice that we now have only one outlier (Iraq). 
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The Calculator 


But what is the estimated model now? 
LOG MIL = 1.308 + 0.909 * LOG GDP 


However, many people don't think in “log units." Let's transform this equation 
(exponentiate each side of the equation): 


10^log mil-10^(-1.308--0.909*1og gdp) 
mil = 107" 909*log( gdp) 


mil = 1071995 +] 02909 0ste%) 


mil = 0.049 * (gdp cap)?” 


We used the calculator to compute 0.049. Type: 
CALC 104(1.308) 
and SYSTAT returns 0.049. 


Example 3 
Residuals and Diagnostics for Simple Linear Regression 


In this example, we continue with the transformations example and save the residuals 
and diagnostics along with the data. Using the saved statistics, we create stem-and-leaf 
plots of the residuals and Studentized residuals. In addition, let’s plot the Studentized 
residuals (to identify outliers in the y space) against leverage (to identify outliers in the 
x space) and use Cook’s distance measure to scale the size of each plot symbol. In a 
second plot, we display the corresponding country names. 


The input is: 


REGRESS 
USE OURWORLD 
LET LOG MIL = L10 (MIL) 
LET LOG_GDP = L10 (GDP_CAP) 
MODEL LOG MIL = CONSTANT + LOG_GDP 
SAVE MYRESULT / DATA RESID 
ESTIMATE 
USE MYRESULT 
CLSTEM RESIDUAL STUDENT 
PLOT STUDENT*LEVERAGE / SYMBOL=4, 2,3 SIZE=cook 
PLOT STUDENT*LEVERAGE / LABEL=COUNTRY$ SYMBOL=4,2,3 
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The output is: 


Stem and Leaf Plot of Variable: 


RESIDUAL, N = 56 


Minimum : -0.644 
Lower Hinge : -0.246 
Median : -0.031 
Upper Hinge : 0.203 
Maximum : 1.216 


-6 

-5 

-4 

=3 

-2 H 65531 
-t 

-0 M 

0 


1 

2 H 009 

3 0113369 
4; 27 
5-1 

6 
7 7 

seo. 


1 


* * * Outside Value: 


1 Cases with missing values excluded from plot 


2 
0.01 002 003 0.04 0.05 0.06 007 008 009 0.10 
LEVERAGE 


Stem and Leaf Plot of Variable: 
STUDENT, N = 56 


Minimum 3 
Lower Hinge : -0. 
Median = -0.091 
Upper Hinge : 0.591 
Maximum : 4.004 
=> 986 
-1 32000 


-0 H 88877766555 


-0 M 443322111000 
0 M 000022344 

0 H 555889999 

1 0223 

i 3 

2.3 


*** Outside Values*** 
4 0 


1 Cases with missing values excluded 
from plot. 


Bor ame 005 004 008 005 097 00 00 01 
LEVERAGE 


In the stem-and-leaf plots, Iraq’s residual is 1.216 and is identified as an Outside Value. 
The value of its Studentized residual is 4.004, which is very extreme for the / 


distribution. 


The case with the most influence on the estimates of the regression coefficients 
stands out at the top left (that is, it has the largest plot symbol). From the second plot, 
we identify this country as Iraq. Its value of Cook’s distance measure is large because 
its Studentized residual is extreme. On the other hand, Ethiopia (furthest to the right), 
the case with the next most influence, has a large value of Cook’s distance because its 
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value of leverage is large. Gambia has the third largest Cook value, and Libya, the 
fourth. 


Deleting an Outlier and Normality Testing 


Residual plots identify Iraq as the case with the greatest influence on the estimated 
coefficients. Let’s remove this case from the analysis and check SYSTAT’s warnings. 


The input is: 


REGRESS 
USE OURWORLD 
LET LOG MIL = L10 (MIL) 
LET LOG_GDP = L10(GDP_CAP) 
SELECT MIL < 700 
MODEL LOG MIL = CONSTANT + LOG GDP 
ESTIMATE/NTEST = KS, SW, AD 


SELECT 
The output is: 

Dependent Variable | LOG MIL 
N ! 55 
Multiple R 1 0.886 
Squared Multiple R 1 0.785 
Adjusted Squared Multiple R | 0.781 
Standard Error of Estimate | 0.306 


Regression Coefficients B = (X'X) *x'Y 


i Std. 
d E: Coefficient Tolerance t 


CONSTANT -1.353 0. 0.000 
LOG GDP | 0.916 0.066 0.886 1.000 13.896 


Regression Coefficients B = (X'X)'X'Y (contd...) 


Effect i p-value 
puc ei er 
CONSTANT | 0.000 
LOG GDP | 0.000 


Analysis of Variance 


Source 1 Mean Squares F-ratio 
----------- * 

Regression | 18.129 1 18.129 193.107 
Residual i 4.976 53 0.094 


Test for Normality 
Test Statistic p-value 


K-S Test (Lilliefors) 
Shapiro-Wilk Test 
Anderson-Darling Test 


EE OES 
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1.763 


Durbin-Watson D Statistic H 
First Order Autocorrelation | 


Information Criteria 

AIC 1 29.931 
AIC (Corrected) | 30.401 
Schwarz's BIC | 35.953 


Now there are no warnings about outliers. From the above results of normaliy tests, the 
assumption of residuals to be normal is satisfied. 


Printing Residuals and Diagnostics 


Let's look at some of the values in the MYRESULT file. We use the country name as 
the ID variable for the listing. 


The input is: 


USE MYRESULT 

IDVAR COUNTRY$ 

FORMAT 10 3 

LIST COOK LEVERAGE STUDENT MIL GDP CAP 


The output is: 


GDP CAP 


* 

Ireland i 0.013 95.833 8970.885 
Austria | 0.023 0.043 -1.011 127,237 13500.299 
Belgium | 0.000 0.044 -0.001 283.939 13724.502 
Denmark + 0.000 0.045 -0.119 269.608  14363.064 
(etc.) 

Libya ; 0.056 0.022 2.348 640.513 4738.055 
Somalia | 0.009 0.072 0.473 8.846 201.798 
Afghanistan | . 0.075 . . 189.128 
(etc.) 


The value of MIL for Afghanistan is missing, so Cook's distance measure and 
Studentized residuals are not available (periods are inserted for these values in the 
listing). 
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Example 4 
Multiple Linear Regression 


In this example, we build a multiple regression model to predict total employment 
using values of six independent variables. The data were originally used by Longley 
(1967) to test the robustness of least-squares packages to multicollinearity and other 
sources of ill-conditioning. SYSTAT can print the estimates of the regression 
coefficients with more "correct" digits than the solution provided by Longley himself 
if you adjust the number of decimal places. By default, the first three digits after the 
decimal are displayed. After the output is displayed, you can use General Linear Model 
to test hypotheses involving linear combinations of regression coefficients. 


The input is: 
REGRESS 
USE LONGLEY 
PLENGTH LONG 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
ESTIMATE 


The output is: 
Eigenvalues of Unit Scaled X'X 


0.000 0.000 
Condition Indices 


1.000 9.142 12.256 25.337 230.424 
Condition Indices 


1048.080 43215.047 
Variance Proportions 


i £ 2 3 4 5 
a A A A a pe 
CONSTANT | 0.000 0.000 0.000 0.000 0.000 
DEFLATOR | 0.000 0.000 0.000 0.000 0,457 
GNP ! 0.000 0.000 0.000 0.001 0.016 
UNEMPLOY | 0.000 0.014 0.001 0.065 0.006 
ARMFORCE | 0.000 0.092 0.064 0.427 0.115 
POPULATN | 0.000 0.000 0.00 0.000 0.010 
TIME 1 0.000 0.000 0.000 0.000 0.000 


11-68 


Chapter 2 


CONSTANT 
DEFLATOR 


Dependent Variable 
N 


Multiple R 

Squared Multiple R H 
Adjusted Squared Multiple R | 0.992 
Standard Error of Estimate | 304.854 


Regression Coefficients B = (X'X)x'Y 


1 Std. 
Effect Coefficient Standard Error Coefficient Tolerance t 
CONSTANT } -3.482E+006 890420.384 0.000 . -3.911 
DEFLATOR | 15.062 84.915 0.046 0.007 0.177 
GNP i -0.036 0.033 71.014 0.001 
UNEMPLOY | -2.020 0.488 -0.538 0.030 
ARMFORCE | 71.033 0.214 -0.205 0.279 
POPULATN ; -0.051 0.226 -0.101 0.003 
TIME i 1829.151 455.478 2.480 0.001 


Regression Coefficients B = (X'X)'X'Y (contd...) 


Effect | p-value 
a a ii *-------- 
CONSTANT ; 0.004 
DEFLATOR } 0.863 
GNP i 0.313 
UNEMPLOY | 0.003 
ARMFORCE | 0.001 
POPULATN | 0.826 
TIME i 0.003 


Confidence Interval for Regression Coefficients 


, 95.0% Confidence Interval 


Effect Coefficient Lower Upper VIF 
micis sc Jane ten desea r CARET e HN 
CONSTANT | -5.497E+006  -1.468E+006 5 
DEFLATOR | -177.029 207.153 135.532 
GNP | -0.112 0.040 1788.513 
UNEMPLOY | -3.125 -0.915 33.619 
ARMFORCE | -1.518 -0.549 3.589 
POPULATN ! -0.563 0.460 399.151 
TIME i 798.788 2859.515 758.981 


Correlation Matrix of Regression Coefficients 
; CONSTANT DEFLATOR GNP UNEMPLOY ARMFORCE 


$ 


CONSTANT ; 


UNEMPLOY | 


TIME 1 
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Correlation Matrix of Regression Coefficients 


| POPULATN TIME 
E Lee premiere seo os 


POPULATN į 1.000 
TIME H 0.388 1.000 


Analysis of Variance 


Source 1 SS df Mean Squares  F-ratio p-value 
pop rante © BERT SERUUM A Eid ci SRE ESS ISTE a 
Regression | 1.842E+008 6 3.070E+007 330.285 0.000 
Residual | 836424.056 9 92936.006 

Durbin-Watson D Statistic 1 2.559 

First Order Autocorrelation | -0.348 


Information Criteria 


AIC | 235.235 
AIC (Corrected) | 255.806 
Schwarz's BIC | 241.416 


SYSTAT computes the eigenvalues by scaling the columns of the X matrix so that the 
diagonal elements of X'X are 1’s and then factoring the X'X matrix. In this example, 
most of the eigenvalues of X'X are nearly 0, showing that the predictor variables 
comprise a relatively redundant set. 

Condition indices are the square roots of the ratios of the largest eigenvalue to each 
successive eigenvalue. A condition index greater than 15 indicates a possible problem, 
and an index greater than 30 suggests a serious problem with collinearity (Belsley, 
Kuh, and Welsh, 1980). The condition indices in the Longley example show a 
tremendous collinearity problem. 

Variance proportions are the proportions of the variance of the estimates accounted 
for by each principal component associated with each of the above eigenvalues. You 
should begin to worry about collinearity when a component associated with a high 
condition index contributes substantially to the variance of two or more variables. This 
is certainly the case with the last component of the Longley data. TIME, GNP, and 
UNEMPLOY load highly on this component. See Belsley, Kuh, and Welsch (1980) for 
more information about these diagnostics. 


Adjusted squared multiple R is 0.992. The formula for this statistic is: 


adj.sq. multiple R = R° — p, *(1-R°) 
(n-p) 
where n is the number of cases and p is the number of predictors, including the 
constant. 
Notice the extremely small tolerances in the output. Tolerance is 1 minus the 
multiple correlation between a predictor and the remaining predictors in the model. 
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The variance inflation factor (VIF) measures how much the variances of the estimated 
regression coefficient are inflated i.e., it identifies the independent variable with 
substantial multicollinearity with other independent variables. It is also defined as the 
reciprocal of tolerance. 


C Gart 
Pe E ER 


R; is the multiple correlation between a predictor and the remaining predictors in the 
model. If one of the regressor variables is nearly linearly dependent on some other 
regressors then R;? will be near unity which implies tolerance to be close to zero and 
VIFs to be large. Large VIFs imply serious problems with multicollinearity. These 
tolerances and V/Fs signal that the predictor variables are highly intercorrelated—a 
worrisome situation. This multicollinearity can inflate the standard errors of the 
coefficients, thereby attenuating the associated F-ratio, and can threaten 
computational accuracy. 

Finally, SYSTAT produces the Correlation matrix of regression coefficients. In the 
Longley data, these estimates are highly correlated, further indicating that there are too 
many correlated predictors in the equation to provide stable estimates. 


Scatterplot Matrix 


Examining a scatterplot matrix of the variables in the model is often a beneficial first 
step in any multiple regression analysis. Nonlinear relationships and correlated 
predictors, both of which cause problems for multiple linear regression, can be 
uncovered before fitting the model. 
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The input is: 


USE LONGLEY 
SPLOM DEFLATOR GNP UNEMPLOY ARMFORCE POPULATN TIME TOTAL / HALF, 
DENSITY=HIST 


The output is: 


f ARMFORCE with the other variables, as 
everal of the predictors. There is also a 
s behavior on ARMFORCE. 


Notice the severely nonlinear relationships o 
well as the near perfect correlations among s 
sharp discontinuity between post-war and 1950' 


Example 5 
Automatic Stepwise Regression 


The following is an example of forward automatic stepping using the LONGLEY data. 
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The input is: 
REGRESS 
USE LONGLEY 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
START / FORWARD 
STEP / AUTO 
STOP 
The output is: 


Stepwise Selection of Variables 


Step Number : 0 
R : 0.000 
R-square : 0.000 


i Std. 
| Effect Coefficient Standard Error Coefficient Tolerance df 


Partial 
Correlation 


Out | Effect Tolerance 


1 230.089 
1.000 1 415.103 
1.000 1 4.729 
1.000 1 3.702 
1.000 1 166.296 
1.000 1 233.704 
Information Criteria 
AIC | 309.619 
AIC (Corrected) | 310.542 
Schwarz's BIC — | 311.164 
Dependent Variable : TOTAL 
Mínimum Tolerance for Entry into Model : 0.000 
Forward Stepwise with Alpha-to-Enter : 0.150 
Forward Stepwise with Alpha-to-Remove : 0.150 
Step Number : 1 
R : 0.984 
R-square : 0.967 
Term Entered : GNP 
i Std. 
In | Effect Coefficient Standard Error Coefficient Tolerance df 
1 Constant KOPP HAUA TAMA Ny UME DA El OL se 
3 | GNP 0.035 0.002 0.984 1.000 1 


3 j| 415.103 0.000 
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i Partial 
Out | Effect Correlation p-value 
bini a canal 
2 | DEFLATOR -0.187 0.017 1 0.473 0.504 
4 | UNEMPLOY -0.638 0.635 1 8.925 0.010 
5 | ARMFORCE 0.113 0.801 1 0.167 0.689 
6 | POPULATN -0.598 0.018 1 7.254 0.018 
7 | TIME -0.432 0.009 1 2.979 0.108 
Information Criteria 
AIC 1 256.857 
AIC (Corrected) | 258.857 
Schwarz's BIC 1 259.175 
Step Number : 2 
R : 0.990 
R-square : 0.981 
Term Entered : UNEMPLOY 
1 Std. 
Effect Coefficient Standard Error Coefficient 
Constant 
| GNP 0.038 0.002 1.071 
| UNEMPLOY -0.544 0.182 -0.145 
I F-ratio p-valu 
1 
3 | 489.314 0.000 
4; 8.925 0.010 
Partial 


Effect 


2 | DEFLATOR -0.073 0.016 1 0.064 
5 | ARMFORCE -0.479 0.486 1 3.580 0.083 
6 | POPULATN -0.164 0.006 1 0.334 0.574 
7 | TIME 0.308 0.002 1 1.259 0.284 
Information Criteria 
AIC 250.494 
AIC (Corrected) 254.131 
Schwarz's BIC 1 253.585 
Step Number 3 
R 0.993 
R-square 0.985 
Term Entered : ARMFORCE 
i Std. 
In | Effect Coefficient Standard Error Coefficient Tolerance 
prm sp LC Todi e t ciii iei een terr eiut 
1 | Constant 
31 GNE 0.041 0.002 1.154 0.318 1 
4 | UNEMPLOY -0.797 0.213 -0.212 0.385 1 
5 | ARMFORCE -0.483 0.255 -0.096 0.486 1 
In i F-ratio p-value 
1} 
3 | 341.684 0.000 
4 | 13.942 0.003 
51 3.580 0.083 
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Partial 
Correlation 


df  F-ratio  p-value 


Information Criteria 


AIC } 248.317 
AIC (Corrected) ! 254.317 
Schwarz's BIC 1 252.180 
Step Number : 4 
R : 0.998 
R-square : 0.995 
Term Entered : TIME 
H 
In | Effect Coefficient Standard Erro: Tolerance df 
L--$--------------L-------2--------------- 
1 | Constant 
3 | GNP -0.040 1 
4 ; UNEMPLOY -2.088 . 1 
5 | ARMFORCE -1.015 0.184 -0.201 0.318 1 
7 | TIME 1887.410 382.766 2.559 0.002 1 
F-ratio p-valu 
0.033 
0.000 
0.000 
0.000 
i Partial 
Out | Effect Correlation Tolerance df  F-ratio p-value 
pd meee nnn eee EM MEME E UL nna 
2 | DEFLATOR 0.143 0.013 1 0.208 0.658 
6 | POPULATN -0.150 0.004 1 0.230 0.642 
Information Criteria 
AIC | 231.655 
AIC (Corrected) | 240.988 
Schwarz's BIC | 236.291 
Dependent Variable | TOTAL 
N i 16 
Multiple R i 0.998 
Squared Multiple R 1 0.995 
Adjusted Squared Multiple R ; 0.994 
Standard Error of Estimate | 279.396 
Regression Coefficients B = (X'X)'x'Y 
i Std. 
Effect Coefficient Standard Error Coefficient Tolerance t 
CONSTANT | -3.599E+006 740632.644 Saini D eeu bei T MP 
GNP i -0.040 0.016 71.137 0.002 
UNEMPLOY | -2.088 -0.556 0.071 
ARMFORCE | -1.015 -0.201 0.318 
TIME i 1887.410 382.766 2.559 0.002 
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Regression Coefficients B = (x'x) !x'Y (contd...) 


CONSTANT | 
GNP 4 
UNEMPLOY i 
ARMFORCE | 
TIME 


Analysis of Variance 


ss df Mean Squares F-ratio p-value 
Reg sion | 842E+008 4 4.604E*007 589.757 0.000 
Residual | 858680.406 11 78061.855 


The steps proceed as follows: 

= Atstep 0, no variables are in the model. GNP has the largest simple correlation and 
F-ratio, so SYSTAT enters it at step 1. Note at this step that the partial correlation, 
Part. Corr., is the simple correlation of each predictor with TOTAL. 
With GNP in the equation, UNEMPLOY is now the best candidate. 


The F-ratio for ARMFORCE is 3.58 when GNP and UNEMPLOY are included in 
the model. 


m SYSTAT finishes by entering TIME. 


In four steps, SYSTAT entered four predictors. None was removed, resulting in a final 
equation with a constant and four predictors. For this final model, SYSTAT uses all 
cases with complete data for GNP, UNEMPLOY, ARMFORCE, and TIME. Thus, when 
some values in the sample are missing, the sample size may be larger here than for the 
last step in the stepwise process (there, cases are omitted if any value is missing among 
the six candidate variables). If you do not want to stop here, you could move more 
variables in (or out) using interactive stepping. 

AIC, AIC (Corrected) and Schwarz's BIC values of the final model are less than the 
corresponding information criteria values of the models fit in previous steps. Thus the 
final model is a better approximation of the true model in comparison to the models in 


previous steps. 


Example 6 
Interactive Stepwise Regression 


Interactive stepping helps you to explore model building in more detail. With data that 
are as highly intercorrelated as the LONGLEY data, interactive stepping reveals the 
dangers of thinking that the automated result is the only acceptable subset model. In 
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this example, we use interactive stepping to explore the LONGLEY data further. That 
is, after specifying a model that includes all of the candidate variables available, we 
request backward stepping by selecting Stepwise, Backward, and Interactive in the 
Regression Estimation tab. After reviewing the results at each step, we use Step to 
move a variable in (or out) of the model. When finished, we select Stop for the final 
model. 


The input is: 


REGRESS 
USE LONGLEY 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
START / BACK 


The output is: 


Stepwise Selection of Variables 


Step Number : 0 
R + 0.998 
R-square : 0.995 


Tolerance 


Constant 

DEFLATOR 15.062 84.915 0.007 1 
GNP -0.036 0.033 0.001 1 
UNEMPLOY -2.020 0.488 0.030 1 
ARMFORCE -1.033 0.214 0.279 1 
POPULATN -0.051 0.226 0.003 1 
TIME 1829.151 455.478 0.001 1 


F-ratio p-value 


031 0.863 
144 0.313 
110 0.003 
252 0,001 
051 0.826 
127 0.003 


Partial 
Correlation 


Information Criteria 


AIC | 235.235 
AIC (Corrected) | 255.806 
Schwarz's BIC | 241.416 


We begin with all variables in the model. We remove DEFLATOR because it has an 
unusually low tolerance and F-ratio value. 


The input is: 
STEP DEFLATOR 


The output is: 


TOTAL 
0.000 
0.150 
: 0.150 


Dependent Variable 

Minimum Tolerance for Entry into Model 
Backward Stepwise with Alpha-to-Enter 
Backward Stepwise with Alpha-to-Remove 


Step Number 
R 


R-square 


Term Removed : 


H Partial 

Out | Effect Correlation 
eal aes e deiecti pma iam 

2 | DEFLATOR 0.059 
Information Criteria 
AIC | 233.291 
AIC (Corrected) | 247. 291 
Schwarz's BIC | 238.699 
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std. 


Coefficient df 


Tolerance 


A MÀ 


p-value 


POPULATN has the lowest F-ratio and, again, a low tolerance. 


The input is: 
STEP POPULATN 
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The output is: 
Step Number : 2 
R A 


R-square : 0.995 
Term Removed : POPULATN 


Std. 
Coefficient Standard Error Coefficient Tolerance df 


-0.040 0.016 -1.137 0.002 1 
-2.088 0.290 -0.556 0.071 1 
71.015 0.184 -0.201 0.318 1 
1887.410 382.766 2.559 0.002 1 
p-value 


1 

31 5.953 0.033 

41 51.870 0.000 

5; 30.496 0.000 

71 24.314 0.000 

Partial 

Out Tolerance df p-value 
0.658 
0.642 


AIC | 231.655 
AIC (Corrected) | 240.988 
Schwarz's BIC | 236.291 


GNP and TIME both have low tolerance values. They could be highly correlated with 
one another, so we will take each out and examine the behavior of the other when we 
do. 


The input is: 


STEP TIME 
STEP TIME 
STEP GNP 


The output is: 


Step Number : 3 

R : 0.993 
R-square : 0.985 
Term Removed : TIME 


Effect Coefficient Standard Error 


* 

| Constant 
i GNP 

1 


0.041 0.002 1.154 
UNEMPLOY -0.797 0.213 -0.212 ES 


ARMFORCE -0.483 0.255 -0.096 0.486 1 
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i Partial 
Out Effect Correlation Tolerance df F-ratio p-value 


2 | DEFLATOR 0.163 1 0.299 0.596 
6 | POPULATN -0.376 0.005 1 1.813 0.205 
7 | TIME 0.830 0.002 1 24.314 0.000 


Information Criteria 


AIC | 248.317 
AIC (Corrected) | 254.317 
Schwarz's BIC | 252.180 
Step Number 4 
R 0.998 
R-square 0.995 
Term Entered : TIME 
Std. 
Standard Error Coefficient Tolerance 
0.016 71.137 0.002 
0.290 -0.556 0.071 
.184 -0.201 0.318 
382.766 2.559 0.002 
31 5.953 0.033 
4| 51.870 0.000 
5| 30.496 0.000 
7| 24.314 0.000 
i Partial 
Correlation Tolerance df F-ratio p-value 
DEFLATOR ing 0.143 0.013 1 0.208 0.658 
POPULATN -0.150 0.004 1 0.230 0.642 
Information Criteria 
AIC 1 231.655 
AIC (Corrected) | 240.988 
Schwarz's BIC 1 236.291 
Step Number : 5 
R : 0.996 
R-square : 0.993 
Term Removed : GNP 
Std. 


ffi Coefficient Standard Error Coefficient Tole 


rance 


1 | Constant 

4 UNEMPLOY -1.470 0.167 -0.391 0.301 
5 | ARMFORCE -0.772 0.184 70.153 0.450 
T TIME 956.380 35.525 1.297 0.257 
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In | F-ratio p-value 


1; 
41 77.320 0.000 
5; 17.67 0.001 
7 | 724.765 0.000 
i Partial 
Out | Effect Correlation Tolerance df F-ratio p-value 
Nice eg tec RES C AE A E DITA a o 
ait} 0.920 
31 . 0.033 
6 1| 0.009 3.768 0.078 


Information Criteria 
AIC 236.576 


AIC (Corrected) | 242.576 
Schwarz's BIC | 240.439 


We are comfortable with the tolerance values in both models with three variables. With 
TIME in the model, the smallest F-ratio is 17.671, and with GNP in the model, the 


smallest F-ratio is 3.580. Furthermore, with TIME, the squared multiple correlation is 
0.993, and with GNP, it is 0.985. Let's stop the stepping and view more information 


about the last model. 
The input is: 
STOP 

The output is: 
Dependent Variable } TOTAL 
N ! 16 
Multiple R | 0.996 
Squared Multiple R | 0.993 
Adjusted Squared Multiple R | 0.991 
Standard Error of Estimate | 332.084 


Regression Coefficients B = (X'X)' lx'y 


Coefficient 


Effect 

CONSTANT -1.797E«006 x 68641.553 WAS 0.000 . * -26.183 
UNEMPLOY ; 71.470 0.167 -0.391 0.301 -8.793 
ARMFORCE į -0.772 0.184 70.153 0.450 74.204 
TIME i 956.380 35.525 1.297 0.257 26.921 


Confidence Interval for Regression Coefficients 


H 95.0% Confidence Interval 
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Effect | Coefficient Lower Upper VIF 
Essi ANM Miles seston amelie E oan h T AE c ur EP Eg 
CONSTANT | -1.797E+006  -1.947E«006  -1.648E+006 A 
UNEMPLOY | -1.470 -1.834 -1.106 3.318 
ARMFORCE | -0.772 -1.173 -0.372 | 2.223 
TIME i 956.380 878.978 1033.782 3.891 


Analysis of Variance 


Source i ss df Mean Squares F-ratio p-value 
lote E, A E o a qr ct CS 
Regression | 1.837E+008 3 6.123E+007 555.209 0.000 
Residual | 1.323E+006 12 110280.062 


Our final model includes only UNEMPLOY, ARMFORCE, and TIME. Notice that its 
multiple correlation (0.996) is not significantly smaller than that for the automated 
stepping (0.998). 


The input is: 


REGRESS 
USE LONGLEY 
MODEL TOTAL-CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
START / BACK 
STEP DEFLATOR 
STEP POPULATN 
STEP TIME 
STEP TIME 
STEP GNP 
STOP 


Example 7 
Testing whether a Single Coefficient Equals Zero 


Most regression programs print tests of significance for each coefficient in an equation. 
SYSTAT has a powerful additional feature—post hoc tests of regression coefficients. 
To demonstrate these tests, we use the LONGLEY data and examine whether the 
DEFLATOR coefficient differs significantly from 0. 
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The input is: 
REGRESS 
USE LONGLEY 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
ESTIMATE / TOL=.00001 
HYPOTHESIS 
EFFECT DEFLATOR 
TEST 
The output is: 
Dependent Variable $ d 
N t 
Multiple R | 0.998 
Squared Multiple R i 0.995 
Adjusted Squared Multiple R | 0.992 
Standard Error of Estimate | 304.854 


Regression Coefficients B = (X'X) X'Y 


H Std. 
Effect | Coefficient Standard Error Coefficient Tolerance t 
890420.384 B . 

84.915 0.046 0.007 0.177 

0.033 71.014 0.001 -1.070 

0.488 -0.538 0.030 -4.136 

0.214 -0.205 0.279 -4,822 

0.226 -0.101 0.003 -0.226 

455.478 2.480 0.001 4.016 


Effect | p-value 
EU Lene sed 
CONSTANT | 0.004 
DEFLATOR ; 0.863 
GNP $ 0.313 
UNEMPLOY | 0.003 
ARMFORCE | 0.001 
POPULATN ; 0.826 
TIME i 0.003 


Analysis of Variance 


Source SS df Mean Squares  F-ratio p-value 

T c MUS RM Pe el s t; Viros fia a 
Regression | 1.842E+008 6 3.070E«007 330.285 0.000 
Residual | 836424.056 9 92936.006 


Test for effect called: DEFLATOR 
Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 
1 Lower Upper 
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Test of Hypothesis 


Source ; ss df Mean Squares F-ratio p-value 
pit cone eq A mmm eer anti ca eiua giai tn inp o m tn t nti rrr 
Hypothesis | 2923.976 1 2923.976 0.031 0.863 
Error | 836424.056 9 92936.006 


Notice that the error sum of squares (836424.056) is the same as the output residual 
sum of squares at the bottom of the ANOVA table. The probability level (0.863) is the 
same also. This probability level (> 0.05) indicates that the regression coefficient for 
DEFLATOR does not differ from 0. 

You can test all of the coefficients in the equation this way, individually, or choose 
All to generate separate hypothesis tests for each predictor or type: 


Example 8 
Testing whether Multiple Coefficients Equal Zero 


You may wonder why you need to bother with testing when the regression output gives 
you hypothesis test results. 


The input is: 


REGRESS 
USE LONGLEY 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
ESTIMATE / TOL=.00001 
HYPOTHESIS 
EFFECT DEFLATOR & GNP 
TEST 


The output is: 
Test for effect called: DEFLATOR and GNP 


A Matrix 


1; 0.000 1.000 
2 | 0.000 0.000 


A Matrix 
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Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 


n ower Upper 
— VERFU e S AS e r Rane 
M f 15.062 84.915 12.939 17.185 
A2 i -0.036 0.033 -0.037 -0.035 


Test of Hypothesis 


Source } 
EA SS A C MI 


Al 2923.976 1 2923.976 0.031 0,863 
A2 106306.259 1 106306.259 1.144 0.313 
^ i 149295.592 2 74647.796 0.803 0.478 
Error | 836424.056 9 92936.006 


Here, the error sum of squares is the same as that for the model, but the hypothesis sum 
of squares is different. We just tested the hypothesis that the DEFLATOR and GNP 
coefficients simultaneously are 0. 

The A matrix printed above the test specifies the hypothesis that we tested. It has 
two degrees of freedom (see the F-ratio) because the A matrix has two rows—one for 
each coefficient. If you know some matrix algebra, you can see that the matrix product 
AB using this A matrix and B as a column matrix of regression coefficients picks up 
only two coefficients: DEFLATOR and GNP. Notice that our hypothesis had the 
following matrix equation: AB = 0, where 0 is a null matrix. 

Ifyou don’t know matrix algebra, don’t worry; the ampersand method is equivalent. 
You can ignore the A matrix in the output. 


Two Coefficients with an A Matrix 


If you are experienced with matrix algebra, however, you can specify your own matrix 
by using AMATRIX. When typing the matrix, be sure to separate cells with spaces and 
press Enter between rows. The following simultaneously tests that DEFLATOR = 0 
and GNP = 0: 


HYPOTHESIS 
AMATRIX [0 
0 


TEST 


You get the same output as above. 


Why bother with AMATRIX when you can use EFFECT? Because in the A matrix, 
you can use any numbers, not just 0’s and 1’s. Here is a bizarre matrix: 


1.0 3.0 0.5 64.3 3.0 2.0 0.0 
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You may not want to test this kind of hypothesis on the LONGLEY data, but there are 
important applications in the analysis of variance where you might. 


Example 9 
Testing Nonzero Null Hypotheses 


You can test nonzero null hypotheses with a D matrix, often in combination using 
CONTRAST or AMATRIX. Here, we test whether the DEFLATOR coefficient 
significantly differs from 30. 


The input is: 


REGRESS 
USE LONGLEY 
MODEL TOTAL = CONSTANT + DEFLATOR + GNP + UNEMPLOY +, 
ARMFORCE + POPULATN + TIME 
ESTIMATE / TOL=.00001 
HYPOTHESIS 
AMATRIX [0 10000 0] 
DMATRIX [30] 
TEST 


The output is: 


A Matrix 


0.000 1.000 0.000 0.000 0.000 


A Matrix 


Null Hypothesis Value for D 
30.000 
Contrast Estimate 


Hypothesis | Estimate (AB-D) Standard Error 95,0% Confidence Interval 
Lower Upper 


84.915 -17.061 -12.815 


Test of Hypothesis 
Source ss df Mean Squares F-ratio p-value 


Hypothesis | 2876.128 1 2876.128 0.031 0.864 
Error | 836424.056 9 92936. 006 
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The commands that test whether DEFLATOR differs from 30 can be performed more 
efficiently using SPECIFY: 
HYPOTHESIS 


SPECIFY DEFLATOR-30 
TEST 


Example 10 
Regression with Ecological or Grouped Data 


If you have aggregated data, weight the regression by a count variable. This variable 
should represent the counts of observations (n) contributing to the ith case. If n is not 
an integer, SYSTAT truncates it to an integer before using it as a weight. The regression 
results are identical to those produced if you had typed in each case. 

We use, for this example, an ecological or grouped data file, PLANTS. 


The input is: 
REGRESS 
USE PLANTS 
FREQ COUNT 
MODEL CO2 = CONSTANT + SPECIES 
ESTIMATE 
The output is: 
Dependent Variable i C02 
N ! 76 
Multiple R i 0.757 
Squared Multiple R 1 0.573 


Adjusted Squared Multiple R | 0.567 
Standard Error of Estimate | 0.729 


Regression Coefficients B = (X'X) x'y 


i Std. 
Effect | Coefficient Standard Error Coefficient 
sens... A 
CONSTANT } 13.738 0.204 0.000 
SPECIES | -0.466 0.047 -0.757 


Regression Coefficients B = (X'X)x'v (contd...) 
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Confidence Interval for Regression Coefficients 
95.0% Confidence Interval 


Effect | Coefficient Lower Upper VIF 
esci Het H a Adnani eL. rc a ra 
CONSTANT | 13.331 14.144 

SPECIES | -0.559 -0.372 1.000 


Analysis of Variance 


Source D ss df lue 


Regression | 52.660 1 52.660 99.223 0.000 
Residual | 39.274 74 0.531 


Example 11 
Regression without the Constant 


To regress without the constant (intercept) term, or through the origin, remove the 


constant from the list of independent variables, REGRESS adjusts accordingly. 


The input is: 


REGRESS 
USE LONGLEY 


MODEL TOTAL = DEFLATOR+ GNP+ UNEMPLOY+ ARMFORCE+ POPULATN + 


TIME 
ESTIMATE 


The output is: 


Model Contains no Constant 


Dependent Variable | TOTAL 
N | 16 

Multiple R | 1.000 
Squared Multiple R i 1.000 
Adjusted Squared Multiple R | 1.000 


Standard Error of Estimate | 475.166 


Regression Coefficients B = (X'X) *x'Y m 
' td. 


Effect | Coeffici 

DEFLATOR ; -52.994 129.545 -0.083 
GNP i 0.071 0.030 0,434 
UNEMPLOY | -0.423 0.418 70.021 
ARMFORCE | -0.573 0.279 -0.024 
POPULATN | -0.414 0.321 -0.745 
TIME 1 48.418 17.689 1.447 


Regression Coefficients B = (X'X)'X'Y (contd...) 


Effect p-value 
DEFLATOR | 0.691 
GNP 0.040 
UNEMPLOY | 0.335 
ARMFORCE | 0.067 
POPULATN ; 0.226 


TIME i 0.021 


t Standard Error Coefficient Tolerance 
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Confidence Interval for Regression Coefficients 


i 95.0% Confidence Interval 


Effect Lower Upper VIF 


DEFLATOR | -52.994 -341.638 235.650 12425.514 
GNP 1 0.071 0.004 0.138 10290.435 
UNEMPLOY | -0.423 71.354 0.507 136.224 
ARMFORCE | -0.573 -1.194 0.049 39.983 
POPULATN | -0.414 71.130 0.302 101193.162 
TIME i 48.418 9.003 87.832 84709.950 


Correlation Matrix of Regression Coefficients 


| DEFLATOR GNP UNEMPLOY ARMFORCE POPULATN 
"— "oaa dos pcd acral i mee oci UM ce Worte a Eb anes ie to teers 
DEFLATOR | 1.000 
GNP i -0.852 1.000 
UNEMPLOY | -0.714 0.830 1.000 
ARMFORCE | -0.289 0.041 0.347 1.000 
POPULATN | 0.644 -0.945 -0.829 0.048 1.000 
TIME 15, 20.762 0.985 0.850 0.009 70.986 


Correlation Matrix of Regression Coefficients 


df Mean Squares F-ratio p-value 
Regression 6.844E+010 6 1.141E*010 50523.396 9.000 
Residual | 2.258E*006 10 225782.260 


Plot of Residuals vs Predicted Values 


RESIDUAL 
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Some users are puzzled when they see a model without a constant having a higher 
multiple correlation than a model that includes a constant. How can a regression with 
fewer parameters predict "better" than another? It doesn't. The total sum of squares 
must be redefined for a regression model with zero intercept. It is no longer centered 
about the mean of the dependent variable. Other definitions of sum of squares can lead 
to strange results, such as negative multiple correlations. If your constant is actually 
near 0, then including or excluding the constant makes little difference in the output. 
Kválseth (1985) discusses the issues involved in summary statistics for zero-intercept 
regression models. The definition used in SYSTAT is Kválseth's formula 7. This was 
chosen because it retains its PRE (percentage reduction of error) interpretation and is 
guaranteed to be in the (0,1) interval. 

How, then, do you test the significance of a constant in a regression model? Include 
a constant in the model as usual and look at its test of significance. 

If you have a zero-intercept model where it is appropriate to compute a coefficient 
of determination and other summary statistics about the centered data, use General 
Linear Model and select Mixture model. This option provides Kválseth's formula 1 for 
R? and uses centered total sum of squares for other summary statistics. 


Example 12 
Regression using SSCP, Covariance or Correlation matrices 


You can regress the data which is in the form of correlation, covariance and SSCP 
matrices by directly inputting the matrix in SYSTAT. Along with it specify the number 
of cases and also dependent and independent variables. The data set used in this 
example is a covariance matrix of NFL data. In this example, we build a multiple 
regression model to predict dependent variable RATING using five independent 


variables. 


We compute a covariance matrix, save it and use it in the regression analysis: 


The input is: 


CORR 
USE NFL 
SAVE NFLCOV 
COVARIANCE ATTEMPTS COMPLETIONS YARDS TDS INTS RATING 
REGRESS 
USE NFLCOV 
MODEL RATING = ATTEMPTS+COMPLETIONS+YARDS+TDS+INTS /N=21 


ESTIMATE 
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The output is: 
Dependent Variable y RATING 
N 121 
Multiple R 1 0.977 
Squared Multiple R | 0.955 
Adjusted Squared Multiple R | 0.940 
Standard Error of Estimate | 0.940 


Regression Coefficients B = (X'X)X'Y 


i Std. 
Effect Coefficient Standard Error Coefficient Tolerance t 
ATTEMPTS 4 -0.021 0.002 -7.121 0.005 729.593 
COMPLETIONS | 0.020 0.004 4.139 0.006 5.6177 
YARDS i 0.001 0.000 2.952 0.008 4.725 
TDS H 0.060 0.012 1.088 0.064 5.027 
INTS i -0.079 0.013 -0.978 0.115 =6.044 
Regression Coefficients B = (X'X) !X'Y (contd...) 
Effect 1 p-value 
prece bry eer 
ATTEMPTS i 0.000 
COMPLETIONS | 0.000 
YARDS i 0.000 
TDS i 0.000 
INTS i 0.000 
Confidence Interval for Regression Coefficients 

95.0% Confidence Interval 

Effect Lower Upper VIF 
ATTEMPTS É 
COMPLETIONS | . 
YARDS i 0.001 0.001 129.628 
TDS H 0.060 0.035 15.560 
INTS i -0.079 -0.106 2 B.695 


Correlation Matrix of Regression Coefficients 
ATTEMPTS COMPLET 


A ME * 
ATTEMPTS i 
COMPLETIONS | 
YARDS i 
TDS i 
INTS i 


Analysis of Variance 


Source 1 SS df Mean Squares  F-ratio 
=-=- A A a 
Regression ¡ 280.002 5 56.000 63.444 0.000 
Residual i 13.240 15 0.883 


In case of correlation matrix, the raw and standardized coefficients are the same. The 
Include constant option is disabled because the matrices are already centered. SYSTAT 
requires the original sample size in order to compute the degrees of freedom. 

If we use the following input the two outputs will have identical results except 
residuals. The coefficients are the same; it takes degrees of freedom from the Cases 
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which is the original sample size. The standardized coefficients (Std. Coefficient) for 
model with constant are the same for covariance and SSCP matrices. 


USE NFL 
REGRESS 
MODEL RATING= CONSTANT+ATTEMPTS+COMPLETIONS+YARDS+TDS+ 
INTS 
ESTIMATE 


Example 13 
Seemingly Unrelated Regression Equations 


This example taken from Judge,et al. (1988), illustrates the efficiency of combining 
two linear models which are contemporaneously correlated, over the individual models 
themselves. The two models and the combined model are given below. The SYSTAT 
data file JUDGEHILL, used in the example is obtained on appending data for the two 
models. It contains two indicator variables X11 and X21 representing the cases 
obtained from the first and second models respectively. X12 and X22 represent the 
market values of a certain product of two different companies with capital stocks X13 
and X23 respectively. The dependent variable Y represents the investment figures for 
the two companies. The data set is fictitious. 


The individual models are 


yi = Xu + Xhi + XisPis a) 


Yi = XB + Xabat Xs Q) 
The combined model is 
y = xn But Xii * Xii + Xn * Xn + Xia 6) 


Since the individual models are correlated, the errors in the combined model are 
correlated. In order to carry out simple linear regression, we should make 
transformations on the data such that the errors become uncorrelated. We first estimate 
the covariance matrix of the combined model and use it for transformation. The 
covariance matrix of the combined model can be derived as the Kronecker product of 
the covariance matrix between the errors of the individual models and the identity 
matrix of order equal to the number of observations in the individual models. An 
estimate of covariance matrix between the errors of the individual models can be 
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obtained using the sample covariance matrix between the residuals of the fitted 
individual models. An adjustment for this matrix is done, since SYSTAT computes 
covariance matrix with a number one less than the number of observations used, in the 
denominator. But the degrees of freedom of the residual sum of squares should be used 
in the denominator for the estimation of covariance matrix. Now use this estimate in 
computing an estimate of the covariance matrix of the combined model. The 
transformed data with uncorrelated errors is obtained by multiplying the inverse of the 
square root matrix of the estimated covariance matrix with the original data. For 
details on the covariance structure of the errors in the combined model, refer Judge,et 
al. (1988). 


The input is: 


USE JUDGEHILL 
SELECT (case < 21) 

REGRESS 

MODEL Y = X11+X12+X13 

SAVE 1.SYZ / RESIDUALS 

ESTIMATE 

SELECT (case > 20) 

REGRESS 

MODEL Y = X21+X22+X23 

SAVE 2.SYZ / RESIDUALS 

ESTIMATE 

MERGE 1.SYZ (RESIDUAL) 2.SYZ (RESIDUAL) 
DSAVE RESID 

CORR 

SAVE COVAR12 

COVARIANCE RESIDUAL 1 RESIDUAL 2 

USE COVAR12 / MAT = SIG MTYPE = NUMERIC 
MAT SIG = SIG(; RESIDUAL 1 RESIDUAL 2 ) 
MAT SIG = SIG*19/17 may xm 
MAT SIG - FOLD(SIG) 

MAT SIG1 - KRON(SIG,I(20)) 

MAT SIGINV = INV(CHOL(SIG1) ) 

USE JUDGEHILL / MAT = DATA MTYPE = NUMERIC 
MAT DATA = DATA(; X11 X12 X13 X21 X22 x23 Y ) 
MAT TRANS DATA = SIGINV*DATA 

MSAVE TRANS DATA 

USE TRANS DATA 

REGRESS 

MODEL Y = X114+X12+X13+X21+X224+X23 
ESTIMATE 
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The output is: 


OLS Regression 


Data for the following results were selected according to 
SELECT (case « 21) 


Model Contains no Constant 


Dependent Variable IT 
N 1 20 
Multiple R ; 0.976 
Squared Multiple R i 0.952 
Adjusted Squared Multiple R | 0.946 
Standard Error of Estimate | 29.324 


Regression Coefficients B = (X'X)X'Y 


H Std. 
Effect | Coefficient Standard Error Coefficient Tolerance t 
aeai pm renga Ad MED 9r ad we eE es PR e 
x11 i 5.279 32.997 0.043 0.039 0.160 
x12 H 0.024 0.016 0.379 0.041 1.441 
x13 i 0.156 0.027 0.593 0.268 5.777 


Regression Coefficients B = (X'X) !X'Y (contd...) 


kl 
1 
s 
ry 

[^ 
E 
o 


Effect 
+ 


Analysis of Variance 


Source p-value 
Regression | 290617.309 3 96872.436 112.652 0.000 
Residual į 14618.730 17 859.925 

Durbin-Watson D Statistic 1 1.496 

First Order Autocorrelation | 0.238 


Information Criteria 

AIC 1 196.644 
AIC (Corrected) | 199.311 
Schwarz's BIC | 200.627 
Residuals have been saved. 
OLS Regression 


Data for the following results were selected according to 
SELECT (case » 20) 


Model Contains no Constant 


Dependent Variable íY 
N | 20 
Multiple R i 0.970 
Squared Multiple R | 0.941 
Adjusted Squared Multiple R ; 0.934 
Standard Error of Estimate | 13.652 
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Regression Coefficients B = (X'X) X'Y 


{ Std. 
Coefficient Standard Error Coefficient Tolerance t 


Effect 


x21 i -0.550 10.718 “ 4 
x22 i 0.062 0.021 0.845 0.043 2.964 
x23 i 0.073 0.075 0.147 0.151 0.970 


Regression Coefficients B = (X'X)'!X'Y (contd...) 


SS df Mean Squares  F-ratio p-value 


Regression | 50730.973 16910.324 90.729 0.000 
Residual 1.3168.523 17 186.384 
Durbin-Watson D Statistic | 2.107 


First Order Autocorrelation | -0.066 
Information Criteria 


AIC | 166.063 
AIC (Corrected) | 168.730 
Schwarz's BIC | 170.046 


Residuals have been saved. 


OLS Regression 
Model Contains no Constant 


Dependent Variable íY 
N | 40 
Multiple R i 0.956 
Squared Multiple R 1 0.915 
Adjusted Squared Multiple R | 0.902 
Standard Error of Estimate ¡| 1.003 


Regression Coefficients B = (X'X)"x'y 


Effect | 


: 
1 
xi. | 
^ 
1 
i 


Regression Coefficients B = (X'X)'!X'Y (contd...) 


1 
Effect | p-value 


xil i 0.930 
x12 H 0.115 
x13 1 0.000 
X21 i 0.748 
x22 i 0.002 
x23 ; 0.258 
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Analysis of Variance 


Source ] SS df Mean Squares  F-ratio p-value 
Regression | 6. 6 61.160 60.799 0.000 
Residual i . 34 1.006 

Durbin-Watson D Statistic i 2.133 

First Order Autocorrelation ¡ -0.082 


Information Criteria 


AIC i 121.251 
AIC (Corrected) | 124.751 
Schwarz's BIC 1 133.073 


The output displayed contains the analysis of three simple linear regressions carried 
over the models (1), (2) and the transformation of the model (3) respectively. The 
estimate of o° is given by the Residual Mean Squares. For the first model the estimate 
is 859.925 and for the second model it is 186.384. Similarly for the transformed model 
the estimate is 1.006, which is much better than those of the first two models. 


Example 14 
Prediction of New Observations 


SYSTAT predicts values of the dependent variable for given values of the independent 
variables, along with its standard error, upper and lower confidence and prediction 
limits. You can input the given values through a .syz file. The names of the variables 
should be the same in both original and new files. Do not input values for the dependent 
(response) variable. In the data file NFL we predict the response variable RATING for 
new values of ATTEMPTS COMPLETIONS YARDS TDS and INTS. The new values 
for prediction are in file NEWNFL. 


The input is: 


REGRESS 
USE NFL 
MODEL RATING = CONSTANT+ATTEMPTS+COMPLETIONS+YARDS+TDS+, 
INTS 


ESTIMATE 
PREDICT NEWNFL 
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The output is: 
Dependent Variable 1 BIRD 
N H 
Multiple R 1 0.977 
Squared Multiple R 1 0.955 
Adjusted Squared Multiple R | 0.940 
Standard Error of Estimate | 1.050 


Regression Coefficients B = (X'X)'X'Y 


y Std. 
Effect Coefficient Standard Error Coefficient Tolerance t 
CONSTANT H 84.346 0.710 0.000 . 118.823 
ATTEMPTS. H 70.021 0.002 77.121 0.005 -9.593 
COMPLETIONS | 0.020 0.004 4.139 0.006 5.677 
YARDS H 0:001 1 0.000 2.952 0.008 4.725 
TDS H 0.060 0.012 1.088 0.064 5.027 
INTS å 70.079 0.013 -0.978 0.115 76.044 
Regression Coefficients B = (X'X)' X'Y (contd...) 
i 
Effect 1 p-value 
+ 
i 
Confidence Interval for Regression Coefficients 
1 95.0% Confidence Interval 
Effect | Coefficient Lower Upper VIF 
CONSTANT 82.833 85.859 d 
ATTEMPTS 70.025 -0.016 . 183.065 
COMPLETIONS 0.012 0.027 176.600 
YARDS 0.001 0.002 129.628 
TDS 0.035 0.086 15.560 
INTS -0.106 70.051 8.695 
Analysis of Variance 
Source i lue 
Regression | 350.002 5 70.000 63.444 0.000 
Residual i 16.550 15 1.103 
New Values 
ATTEMPTS COMPLETIONS YARDS TDS INTS. 
5100.000 4122.000 54213.000 250.000 201.000 
2333.000 2431.000 26754.000 231.000 198.000 
4532.000 1342.000 65742. 000 145.000 114.000 
2234.000 1675.000 54897.000 121.000 176.000 
5467.000 3421.000 15478 .000 117.000 123.000 
3249.000 5643.000 38765.000 187.000 127.000 
4167.000 2318.000 18762.000 327.000 149.000 
Prediction of New Values 
Predicted Standard Error 95.0% Confidence Interval 95.0% Prediction Interval 


Lower Upper Lower 
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123.387? 4.674 113.425 133.350 113.176 
114.011 4.129 105.209 122.813 104.929 
93.926 11.292 69.857 117.994 69,753 
129.114 8.880 110.186 148.041 110.054 
54.624 5.301 43.326 65.923 43.106 
175.881 12.266 149.737 202.025 149.641 

4.048 3.899 65.739 82.358 65.442 


Confidence limits are limits for a mean response at a level of predictor values, whereas 
prediction limits are limits for the response of a randomly selected unit from the 


population at a certain level of predictor values. 
You can also save the new predicted values along with it the new set of values of 


independent variables in the model. 


The input is: 


REGRESS 


USE NFL 
MODEL RATING- CONSTANT+ATTEMPTS+COMPLETIONS+YARDS+TDS+ INTS 


ESTIMATE 
SAVE PREDICT/PREDICT, NEWDATA 
PREDICT NEWNFL 


Example 15 
Ridge Regression Analysis 


In this example, we build a multiple regression model to predict dependent variable 
TOTAL using values of six independent variables - DEFLATOR, GNP, UNEMPLOY, 
ARMFORCE, POPULATN, TIME. The data were originally used by Longley (1967) 
to test the robustness of least-squares packages to multicollinearity and other sources 
of ill-conditioning. 
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The input is: 


RIDGEREG 
USE RLONGLEY 
MODEL TOTAL = DEFLATOR+GNP+UNEMPLOY+ARMFORCE+POPULATN+TIME 
ESTIMATE/ LMIN=.2 LMAX=.6 LSTEP=.1 


The output is: 
Hoerl-Kennard-Baldwin (HKB) Estimator : 0.000 
Lawless & Wang (LW) Estimator : 0.003 
Minimum Value of Generalized Cross Validation (GCV) is at Lambda : 0.600 


Standardized Ridge Coefficients 


LAMBDA DEFLATOR GNP UNEMPLOY ARMFORCE POPULATN TIME 
0.200 0.241 0.276 -0.115 0.011 0.230 0.251 
0.300 0.229 0.257 70.075 0.034 0.221 0.234 
0.400 0.220 0.243 70.048 0.048 0.213 0.223 
0.500 0.213 0.232 70.029 0.057 0.206 0.214 
0.600 0.206 0.223 -0.015 0.063 0.200 0.207 


0.200 -320.392 0.010 -0.004 0.001 0.116 0.185 
0.300 -295.796 0.009 -0.003 0.002 0.112 0.173 
0.400 -278.868 0.009 -0.002 0.002 0.108 0.164 
0.500 -265.733 0.008 -0.001 0.003 0.104 0.158 
0.600 -254.844 0.008 -0.001 0.003 0.101 0.152 


LAMBDA DEFLATOR GNP UNEMPLOY Al 


Ridge regression estimators have a bias, but a smaller mean square error than that of 
ordinary least-squares estimates. SYSTAT produces estimates of the bias vector for all 
lambdas and covariance matrix of standardized ridge regression coefficients for a 
given lambda or the first value from a set of lambda values. 
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Ridge Regression Parameters 


Sepe pene pP p= 


Ridge Factor 
h 


Example 16 
Bayesian Regression 


To illustrate the Bayesian regression, we use the data related to the Cobb-Douglas 
production function (Judge et al., 1988). 


The Cobb-Douglas production function is given by: 


Q= al’ K” exp(é) 


where Q, L, and K represent the output, ‘labor’, and ‘capital invested! respectively. 
When a logarithmic transformation is used for Qt, L, and K, we obtain the linear 


regression model: 


Y = p, +X, +X: +E 


where Y=Log(Q), Po = Log(a). X; = Log(L), and X; = Log(K). 
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The data set consists of 20 observations and the purpose here is to study the effect 
of labor and capital on the output Y. To fit a Bayesian regression model to this data, 
we have to specify the parameters of the prior distribution. 

The mean vector and covariance matrix of the (multivariate) Normal prior 
distribution of the regression coefficients: 


b, = (5.0, 0.5, 0.5) 
and 
924.0 0.0 0.0 
Vo=| 0 0.3695 -0.349 
0  -0.349 0.3695 


The parameters of the gamma prior distribution for the variance: 
Scale = 4.0 and Shape- 0.0754. 


The input is: 


BAYESIAN 
USE COBDOUG 
MODEL Y=CONSTANT +X1+X2 
SAVE BAYOUT1 / COEFFICIENTS 
ESTIMATE / MEAN -[5;0.5;0.5] VAR = [924 0 0;0 0.3695, 
-0.349;0 -0.349 0.3695] SCALE = 4 SHAPE = 0.0754 


The output is: 

Normal Prior Mean 

5.000 0.500 0.500 

Normal Prior Covariance Matrix 

924.000 0.000 0.000 
0.000 0.370 -0.349 
0.000 -0.349 0.370 

Gamma Prior Parameters 


Scale Parameter 4.000 
Shape Parameter 0.075 
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Bayesian Estimate of Regression Coefficients and Credible Intervals 
95% Credible Interval 


Effect ! Coefficient Standard Error Lower Upper 
mi Sian jacana eee HSE o emma 
CONSTANT | 10.028 0.101 9.830 10.227 
xi H 0.476 0.077 0.325 0.627 
x2 i 0.548 0.083 0.384 0.711 


Bayesian Estimate of Error Variance 


Error Varianci t 


| 0.082 


Parameters of Posterior Distribution 
Conditional Distribution of Regression Coefficient given Sigma follows Multivariate Normal wi 
Mean Vector 
10.028 0.476 0.548 
Covariance Matrix 
Sigma^2 * 
0.123 -0.035 -0.014 
-0.035 0.071  -0.059 
-0.014 -0.059 0.083 
Marginal Distribution of Regression Coefficient follows Multivariate Students T with 
Mean Vector 
10.028 0.476 0.548 
Covariance Matrix 
0.010 -0.003  -0.001 
-0.003 0.006 -0.005 
-0.001 -0.005 0.007 
Marginal Distribution of (1/Sigma)^2 is 
Gamma with 


Scale Parameter 24.000 
Shape Parameter 0.076 
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Prior and Posterior Densities of CONSTANT 


-100 5 0 5 100 


Prior and Posterior Densities of Coefficient of X1 
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Prior and Posterior Densities of Coefficient of X2 


Density 


1». 
B PRIOR 
E POSTERIOR 
05 3 4 1 3 5 
x2 
Prior and Posterior Densities of 1/(Sigma”2) 
0.15 
0.10 
2 
2 
o 
à 
0.05 
E PRIOR 
Qe 
995 10 30 40 50 
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Computation 


Algorithms 


RIDGEREG module uses the ridge regression estimator proposed by Hoerl and 
Kennard (1970). 
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Chapter 


Linear Models II: Analysis of 
Variance 


Leland Wilkinson and Mark Coward (revised by Sayyad Nisar Badashah and Amol Patil) 


SYSTAT handles a wide variety of balanced and unbalanced analysis of variance 
designs (Speed et al., 1978). The Analysis of Variance (ANOVA) procedure includes 
all interactions in the model and tests them automatically. Analysis of covariance and 
the repeated measures designs are a part of the ANOVA feature. Once you have 
estimated your ANOVA model, it is easy to test the post hoc pairwise differences in 
means or to test any contrast across cell means, including simple effects. 

SYSTAT offers three tests for checking normality: Kolmogorov-Smirnov 
(Lilliefors), Anderson-Darling, and Shapiro-Wilk test; and Levene's test for 
homogeneity of variances. You can select any of the three types of sum of squares, 
Type I, Type II, and Type III, for the analysis. 

The ANOVA module provides fifteen tests for pairwise comparisons based on the 
structure of data and the error rate to be controlled. The pairwise comparison tests are 
commonly named as post hoc tests; here tests are determined based on the 
assumptions on variance, viz., equal or unequal variances. One can use post hoc tests 
after fitting the ANOVA model to check the differences between pairs of means. 

The General Linear Model (GLM) procedure is used for randomized block designs 
(Kutner et al., 2004), incomplete block designs, fractional factorials, Latin square 
designs (Cochran and Cox, 1957; John, 1971), and analysis of covariance with one or 
more covariates. GLM also includes repeated measures, split plot, and crossover 
designs. It includes both univariate and multivariate approaches to repeated measures 
designs (Bartlett, 1947; Morrison, 2004). 

For both ANOVA and GLM, group sizes can be unequal for the combinations of 
grouping factors; but for repeated measures designs, each subject must have complete 
data. You can use numeric or character values to code the grouping variables. 
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You can store results of the analysis (predicted values and residuals) for further 
study and graphical display. In ANCOVA, you can save adjusted cell means. AIC, AIC 
(Corrected) and Schwarz's (1978) BIC values are also provided for each fitted model 
(Burnham and Anderson, 2003). For more information on AIC and Schwarz's BIC in 
SYSTAT refer to the section “Variable Selection“ on page 15 in the chapter on Linear 
Models in Statistics II. 

Resampling procedures are available in this feature. 


Analysis of Variance in SYSTAT 


Analysis of Variance: Estimate Model Dialog Box 


To obtain an analysis of variance, from the menus choose: 


Analyz 
Analysis of Variance (ANOVA) 
Estimate Model... 
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Available variable(s]: 
ed 


A 


YIELD(T) 
YIELD(2) 
YIELD(3) 
GROUP — 
A$ 

B$ 


[ YIELD(1) 
YIELD(2) 


Factor(s] 


Dependent(s). The variable(s) you want to examine. The dependent variable(s) should 
be continuous and numeric (for example, INCOME). 


Factor(s). One or more categorical variables (grouping variables) that split your cases 


into two or more groups. 


m Missing value. Includes a separate category for cases with a missing value for the 


variable(s) identified with Factor. 


Covariate(s). A covariate is a quantitative independent variable that adds unwanted 
variability to the dependent variable. An analysis of covariance (ANCOVA) adjusts or 
removes the variability in the dependent variable due to the covariate (for example, 
variability in cholesterol level might be removed by using AGE as a covariate). 


Save. You can save residuals and other data to a new data file. The following 


alternatives are available: 


m Adjusted. Saves adjusted cell means from analysis of covariance. 
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m Adjusted/Data. Saves adjusted cell means plus all of the variables in the working 
data file, including any transformed data values. 


Coefficients. Saves estimates of the regression coefficients. 
Model. Saves statistics given in Residuals and the variables used in the model. 
Partial. Saves partial residuals. 


Partial/Data. Saves partial residuals plus all the variables in the working data file, 
including any transformed data values. 


m Residuals. Saves predicted values, residuals, Studentized residuals, leverages, 
Cook’s D, and the standard error of predicted values. Only the predicted values and 
residuals are appropriate for ANOVA. 


m Residuals/Data. Saves the statistics given by Residuals plus all of the variables in 
the working data file, including any transformed data values. 


Repeated Measures 


Ina repeated measures design (Cochran and Cox, 1957), the same variable is measured 
several times for each subject (case). A paired-comparison z test is the most simple 
form of a repeated measures design (for example, each subject has a before and after 
measure). 

SYSTAT derives values from your repeated measures and uses them in analysis of 
variance computations to test changes across the repeated measures (within subjects) 
as well as differences between groups of subjects (between subjects). Tests of the 
within-subjects values are called polynomial test of order 1, 2,., up to k, where k is one 
less than the number of repeated measures. The first polynomial is used to test linear 
changes; perform the repeated responses increase (or decrease) around a line with a 
significant slope. The second polynomial tests whether the responses fall along a 
quadratic curve, and so on. 

To perform repeated measures analysis, click the Repeated Measures tab in 
Analysis of Variance: Estimate Model dialog box. 


Ir-111 


Linear Models II: Analysis of Variance 


The following options are available: 


Perform repeated measures analysis. Treats the dependent variables as a set of 
repeated measures. Optionally, you can assign a name for each set of repeated 
measures, specify the number of levels, and specify the metric for unevenly spaced 
repeated measures. 

m Name. Name that identifies each set of repeated measures. 

m Levels. Number of repeated measures in the set. For example, suppose you have 
three dependent variables that represent measurements at different times, the 
number of levels is 3. 

m Metric. Metric that indicates the spacing between unevenly spaced measurements. 
For example, if measurements were taken at the third, fifth, and ninth weeks, the 
metric would be 3, 5, 9. 
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Options 


To specify the options, click the Options tab in the Analysis of Variance: Estimate 
Model dialog box. 


FA Analysis of Variance: Estimate Model 


]Kolmogotov.Smimov — [ ) Shapiro-Wilk 
| [JAndersor-Daring 
+ Equality of variances tests — — 


Levene 


Sums of squares 

| © Type |: Sequential 

O Type Il: Partially sequential 
| © Type Ill: Adiusted 


Assumptions check. This provides options to check the basic assumptions of ANOVA. 


Normality tests. You can use the following normality tests to check the basic statistical 

assumption of ANOVA, normality of residuals: 

m Kolmogorov-Smirnov. It is a nonparametric test used for large samples. It is 
applied to continuous distributions and gives greater importance to the 
observations in the center than those at the tails. 

m Shapiro-Wilk. The test provides the Shapiro-Wilk test statistic and p-value for 
residuals: the smaller the p-value, the worse is the fit. 
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m Anderson-Darling. The Anderson-Darling test is a standard goodness of fit test. It 
gives greater importance to the observations in the tails than those at the center. 


Equality of variances tests. You can use the following equality of variance test to check 
the homogeneity of variances across all levels of the factors: 


m Levene’s. The Levene's test is less sensitive than the Bartlett test to departures from 
normality. 


Sum of squares. For the model, you can choose a particular type of sum of squares. 
Type III is the one most commonly used and is the default. 


m Type I: Sequential. Uses type I sum of squares for the analysis. 
m Type II: Partially sequential. Uses type Il sum of squares for the analysis. 


m Type III: Adjusted. Uses type III sum of squares for the analysis. This is the 
default. 


Analysis of Variance: Hypothesis Test Dialog Box 


Contrasts are used to test relationships among cell means. Use Specify or Contrast to 
define contrasts involving two or more means—for example, contrast the average 
responses for two treatment groups against that for a control group; or test if average 
income increases linearly across cells ordered by education (dropouts, high school 
graduates, college graduates). The coefficients for the means of the first contrast might 
be (1, 1, 2) for a contrast of /* Treatment A plus 1* Treatment B minus 2 * Control. 
The coefficients for the second contrast would be (1, 0, 1). (For more information, see 
Wilkinson, 1975). 

The ANOVA model must be estimated before any hypothesis tests can be 
performed. To define contrasts among the cell means, from the menus choose: 
Analyze 


Analysis of Variance (ANOVA) 
Hypothesis Test... 
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Analyze:Analysis of Variance (ANOVA):Hypothesis Test 


Main | y Contrast 


Hypothesis: [Efecto IB 
Available effects Selected effects) 
| GROUP. 

Constant 


Contrasts can be defined across the categories of a grouping factor or across the levels 
of a repeated measure. 


Selected effect(s). Select one or more effects you want to test. 


Hypothesis. Select the type of hypothesis. The following choices are available: 

= Model. Tests for the coefficients of the model. This is the default 

W All. Select to test all main effects and interactions. 

W Effects. Select one or more effects you want to test. 

W Specify. Select Specify to use Specify tab. 

Within. Use when specifying a contrast across the levels of repeated measures factor. 


Select the name assigned to the set of repeated measures in the Repeated Measures 
tab. 
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Specify 


To specify coefficients for hypothesis tests, select Specify option of Hypothesis in the 
Analysis of Variance: Hypothesis Test dialog box. 


Analyze:Analysis of Variance (ANOVA):Hypothesis Test 


Example 
uu UL 
2*A[1] « A[2] 


To specify coefficients for a hypothesis test, use cell identifiers. The common 

hypothesis tests include contrasts across marginal means or tests of simple effects. For 
a two-way factorial ANOVA design with DISEASE (four categories) and DRUG (three 
categories), you could contrast the marginal mean for the first level of drug against the 


third level by specifying: 


DRUG[1] = DRUG[3] 


Note that square brackets enclose the value of the category (for example, for 
GENDERS, specify GENDERS[male]). For the simple contrast of the first and third 
levels of DRUG for the second disease only: 


DRUG[1] DISEASE[2] = DRUG[3] DISEASE [2] 
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The syntax also allows statements like: 


-3*DRUG[1] - 1*DRUG[2] + 1*DRUG[3] + 3*DRUG[4] 


You have two error term options for hypothesis tests: 
m Pooled. Uses the error term from the current model. 


W Separate. Generates a separate variance error term. 


Contrast 


To specify contrasts, select an effect under the Effect option of Hypothesis and click 
the Contrast tab in the Analysis of Variance: Hypothesis Test dialog box. 


Analyze:Analysis of Variance (ANOVA):Hypothesis Test 


Custom example 
3413 
13-31 


O Adjacent Difference 
O Helmert 

O Reverse Helmert 
© Polynomial 


Order, 


[Peay 


Metric 
JEJEJ 


Contrast generates a contrast for a grouping factor or a repeated measures factor. 
SYSTAT offers eight types of contrasts: 
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Custom. Enter your own custom coefficients. If your factor has, say, four ordered 
categories (or levels), you can specify your own coefficients, such as -3 -1 1 3, by 
typing these values in the Custom text box. 


Adjacent difference. Compare each level with its adjacent level of the selected factor. 
Helmert. Compares the mean of each level of the selected factor with the mean of the 
succeeding levels. 

Reverse Helmert. Compares the mean of each level of the selected factor with the 

mean of the previous levels. 

Polynomial. Generates orthogonal polynomial contrasts (to test linear, quadratic, cubic 

trends across ordered categories or levels). 

m Order. Enter 1 for linear, 2 for quadratic, and so on. 

m Metric. Use Metric when the ordered categories are not evenly spaced. For 
example, when repeated measures are collected at weeks 2, 4, 8, enter 2,4, 8 as the 
metric. 

Deviation. The deviation contrast compares the mean of the dependent variable for 

each level of the selected categorical variable (except a reference level) to the overall 

mean (grand mean) of the dependent variable. 

Simple. The simple contrast compares each level of the selected factor against the 

specified reference level. This type of contrast is useful when there is a control group. 

You can choose any level or category as the reference. 


Sum. In a repeated measures ANOVA, total the values for each subject. 


Analysis of Variance: Pairwise Comparisons Dialog Box 


After fitting the model, one can find the treatment pairs which are significantly 
different, or form several homogeneous sets of treatments with their respective p-value 
by using several multiple comparison tests (mct) offered by SYSTAT under equal or 


unequal variance assumptions. 
To open Pairwise Comparisons dialog box, from menus choose: 
Analyze 


Analysis of Variance (ANOVA) 
Pairwise Comparisons... 
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Analyze: Analysis of Variance (ANOVA): Pairwise Comparisons 


Main | Emor Teml my 
Avaliable effect(s} - Groups: 22621 Ins 
JOBCAT | | JOBCAT 
== 
| 
Tests 
© Equal variances 
[V] Tukey E Duncan [V] Dunnett 
E Bontenoni Orewa © 2-sided 
DFishers LSO [7] Hochberg's GT2 O Less than control 
F Sidak [7] Gabriel O Greater than control 
[7] Schefte [7] Student-Newman-Keuls Cow [7 — — 
C Tukey's b 
| O Unequal variances 
Tamhane's T2 y | Games-Howell Dunnett's T3 


OEC i 


Groups. Select the variable that defines the groups. 


Tests. There are several post hoc tests to compare the means of the dependent variable 
for the selected grouping variable. 


Equal variances. Tests in this group assume equality of variances across all levels of 
the grouping variable. 


m Tukey. Uses the Studentized range distribution to make all pairwise comparisons. 
This is the default. 


m Bonferroni. Uses Student's / statistic. It sets the family-wise error rate as 
(1- Confidence)/(Total number of comparisons). 


m Fisher's LSD. Equivalent to multiple t-tests between all pairs of groups. The 
disadvantage of this test is that no attempt is made to adjust the observed 
significance level for multiple comparisons. 


Sidak. Uses Student's / statistic for pairwise multiple comparisons. 


Scheffé. The significance level of Scheffé’s test is designed to allow all possible 
linear combinations of group means to be tested, not just pairwise comparisons 
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available in this feature. The result is that Scheffé's test is more conservative than 
other tests. 

m Tukey'sb. Uses the Studentized range distribution. The critical value is the average 
of the corresponding values for the Turkey's HSD test and the Student-Newman- 
Keuls (S-N-K) test. 

m Duncan. Uses Studentized range distribution. It yields homogeneous subsets of 
group levels. 

m R-E-G-W Q. Ryan-Einot-Gabriel-Welsch Q test is a modification ofthe S-N-K test 
where the critical values decrease as the range in the set being considered 
decreases. 

m Hochberg's GT2. Uses the Studentized maximum modulus distribution 

m Gabriel. Uses the Studentized maximum modulus distribution. It is equivalent to the 
GT2 test for balanced ANOVA. 

m Student-Newman-Keuls. Uses the Studentized range distribution. It yields 
homogenous subsets of group levels. 

m Dunnett. The Dunnett test is available only with one-way designs. Dunnett 
compares a set of treatments against a single control mean that you specify. Y ou can 
choose from the following three alternative hypotheses: (a) 2-sided (not equal), 
(b) less than, or (c) greater than the control level. 2-sided is the default. 


Unequal variances. The following tests do not require the homogeneity of variance 

assumption. These tests use the Welsch procedure for determining the denominator 

degrees of freedom. 

m Tamhane's T2. Uses the Student's f distribution.Uses the Sidak inequality to find 
the alpha level. 

m Games - Howell. Uses the Studentized range distribution. 

m Dunnett's T3. Uses the Studentized maximum modulus distribution. 

Confidence. Specify confidence level for pairwise comparisons tests. The default value 

is 0.95. 


Error term 


To specify the error term, click the Error Term tab in the ANOVA: Pairwise 
Comparisons dialog box. 
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Analyze: Analysis of Variance (ANOVA): Pairwise Comparisons 


| Main | Error Tem 


Between-subjects effect(s) 


You can choose one of the following: 

m Model MSE. Uses the mean square error (MSE) from the general linear model that 
you ran. 

= MSE and df. Uses the mean square error term and degrees of freedom that you _ 
specify. Use this option if you know them from a previous model. 


W Between-subjects effect(s). Select this option to use the main effect error term or 
the interaction error term in all the tests. 


Note: Toggling between the command line and GUI is supported in ANOVA, GLM, 
MANOVA, REGRESS, MIXED, LOGIT, LOGLINER, and RSM. That is, if estimation is 
performed through the dialog box then post estimation analysis can be performed 
through commands, and vice-versa. 
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Using Commands 


ANOVA 
USE filename 
CATEGORY varlist/ MISS EFFECT or DUMMY 
DEPEND varlist / REPEAT NAMES 
COVAR varlist 
PLENGTH NONE or SHORT or MEDIUM or LONG 
SAVE filename / ADJUST, COEFFICIENT, MODEL, PARTIAL 
RESID, DATA ‘comment’ 
WORK filename / ADJUST, COEFFICIENT, MODEL, PARTIAL 
RESID, DATA ‘comment’ 
ESTIMATE / NTEST = KS, SW, AD HTEST = LEVENE 
SS = TYPE1 or TYPE2 or TYPE3 QUICK or NOQUICK 
SAMPLE = BOOT(m,n) or SIMPLE(m,n) or JACK 


To use ANOVA for analysis of covariance, insert COVARIATE before ESTIMATE. 


After estimating a model, use HYPOTHESIS to test its parameters. Begin each test with 
HYPOTHESIS and end with TEST. 


HYPOTHESIS 
ALL 
EFFECT varlist or varl*var2 or varl&var2 


WITHIN ‘name’ 
ERROR value (df) or var or varl*var2 or varl&var2 or matrix 


POST grpvar / TUKEY or BONF-n or LSD or SIDAK or SCHEFFE or 
BTUKEY or DUNCAN or QREGW or GT2 or GABR or 
SNK or GH or T2 or T3 or SEPARATE or 
POOLED or DUNNETT - LT or GT or TWO 
CONTROL - 'levelname' 
CONTRAST / ADJDIFF or POLYNOMIAL, ORDER-n METRIC=m, n,... 
or SUM or DEV[c]or SIMPLE[c] or HEL or RHEL 
SPECIFY / POOLED or SEPARATE 
AMATRIX [matrix] 
CMATRIX [matrix] 


DMATRIX [matrix] 
PAIRWISE / BONFERRONI or SIDAK 


TEST / CONFI = n 


Usage Considerations 


Types of data. ANOVA requires a rectangular data file. 
ENGTH SHORT, the output includes an ANOVA table. The MEDIUM 


Print options. If PL 
to the output. LONG adds the estimates of the 


length adds the least-squares means 
coefficients. 
Quick Graphs. ANOVA plots the group means against the groups. . 
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Saving files. ANOVA can save predicted values, residuals, Studentized residuals, 
leverages, Cook's D, standard error of predicted values, adjusted cell means, and 
estimates of the coefficients. 


BY groups. ANOVA performs separate analyses for each level of any BY variables. 
However, for Hypothesis Testing, BY groups does not work. You have to resort to 
Data--> Select Cases commands. 


Case frequencies. You can use a FREQUENCY variable to duplicate cases. 


Case weights. ANOVA uses a WEIGHT variable, if present, to duplicate cases. 


Examples 


Example 1 
One-Way ANOVA 


How does equipment influence typing performance? This example uses a one-way 
design to compare average typing speed for three groups of typists. Fourteen beginning 
typists were randomly assigned to three types of machines and given speed tests. The 
following are their typing speeds in words per minute: 


Electric Plain old Word 


processor 
52 52 67 
47 43 73 
51 47 70 
49 44 75 
53 64 


The data are stored in the SYSTAT data file named TYPING, The average speeds for 
the typists in the three groups are 50.4, 46.5, and 69.8 words per minute, respectively. 
To test the hypothesis that the three samples have the same population average speed, 
the input is: 
ANOVA 
USE TYPING 
CATEGORY EQUIPMNTS 


DEPEND SPEED 
ESTIMATE 
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The output is: 
Dependent Variable | SPEED 
N ; 14 
Multiple R i 0.952 


Squared Multiple R | 0.907 


Analysis of Variance 


Source | Type III SS df Mean Squares F-ratio p-value 
EQUIPMNTS | 1469.357 2 134.679 53.520 0.000 
Error | 151.000 ' 11 13.727 


Least Squares Means 


EQUIPMNTS 


For the dependent variable SPEED, SYSTAT reads 14 cases. The multiple correlation 
(Multiple R) for SPEED with the two design variables for EQUIPMNTS is 0.952. The 
square of this correlation (Squared multiple R) is 0.907. The grouping structure 
explains 90.796 of the variability of SPEED. 

The layout of the ANOVA table is standard in elementary texts; you will find 
formulas and definitions there. F-ratio is the Mean-Square for EQUIPMNTS divided 
by the Mean-Square for Error. The distribution of the F-ratio is sensitive to the 
assumption of equal population group variances. The p-value is the probability of 
exceeding the F-ratio when the group means are equal. The p-value printed here is 
0.000, so it is less than 0.0005. If the population means are equal, it would be very 
unusual to find sample means that differ as much as these—you could expect such a 
large F-ratio fewer than five times out of 10,000. 
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The Quick Graph illustrates this finding. Although the typists using electric and plain 
old typewriters have similar average speeds (50.4 and 46.5, respectively), the word 
processor group has a much higher average speed. 


Pairwise Mean Comparisons 


An analysis of variance indicates whether (at least) one of the groups differs from the 
others. However, you cannot determine which group(s) differs based on ANOVA 
results. To examine specific group differences, use post hoc tests. 

In this example, we use the Bonferroni method for the typing speed data used in the 
one-way ANOVA example. As an aid in interpretation, we order the equipment 
categories from least to most advanced. 


The input is: 


HYPOTHESIS 
POST EQUIPMNT$/ BONF 
TEST 


The output is: 

Post Hoc Test of SPEED 

Using least squares means. 

Using model MSE of 13.727 with 11 df. 


Bonferroni Test 


EQUIPMNTS (i) EQUIPMNTS (3) Difference p-value 95.0% Confidence Interval 

Upper 
electric plain old 3.900 0.435 -3.100 10.909 
electric word process -19.400 0.000 =26.008 -12.792 
plain old word process -23.300 0.000 -30.309 -16.29 


In the first and second rows, you can read differences in average typing speed and 
corresponding 95% confidence intervals for the group using plain old typewriters. In 
the third column, row one, you see that the average is 3.9 words per minute fewer than 
those using electric typewriters; but in the second row, you see that the average is 23.3 
minutes fewer than the group using word processors. To see whether these differences 
are significant, look at the probabilities in the fourth column. 

The probability associated with 3.9 is 0.435, so you are unable to detect a difference 
in performance between the electric and plain old groups. The probabilities in the 
second and third row are both 0.00, indicating that the word processor group averages 
significantly more words per minute than the electric and plain old groups. 
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Contrasts 


To compare the differences in the means of levels of a single factor, you can use 
SYSTAT's CONTRAST command. In this example, suppose you want to contrast 
'electric' equipment against 'word processor' equipment, you can use the following 
commands. 


The input is: 


PLENGTH LONG 
HYPOTHESIS 
EFFECT EQUIPMNT$ 
CONTRAST [-1 0 1] 
TEST 


The output is: 
Test for effect called: EQUIPMNTS 


A Matrix 


-2.000 


0.000 
Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 
Lower Upper 
A diee s 719.400 2.343 19.341 19.459 


Test of Hypothesis 


Source H Mean Squares F-ra 

ain nare E. + 
Hypothesis | 940.900 1 940.900 68.542 
Error | 151.000 11 13.727 


The model for the above analysis is: 


yy = pntat; 


where ¡=1,2,3 and ¡=1(1) nj 
The parameters p, 01, 01, and a; satisfy the following condition: 
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The contrast in this example is coded as [-1 0 1]. It imposes the restriction on the levels 
of equipment, which is 
03-0, -0 
Using the above assumption, we get 
a, - a5) - a4 - 0 
which, in turn, reduces to 
-2a, -05- 0 
i.e. 0u — 2a) - %2 = 0 
Now, look at the term A matrix in the output created by the first and third levels of the 
equipment. The A matrix for the above model is [0 -2 -1]. Notice that the value 0 
corresponds to the constant term and -2 and -1 for the first and second design variables 
in the model. The Contrast estimate of the A matrix is 19.4, the corresponding S.E. is 
5.4909 and 95% confidence intervals are 7.3145, 31.4854. 

The F-ratio for testing the contrast is 68.542 (p-value < 0.0005). Thus you can 
conclude that there is a significant difference between the first and third levels of 
equipment. 


Similarly the contrast a, — 2a + a3 = 0 can be tested by defining the A matrix as 
[0 0 -3]. 


Example 2 
ANOVA Assumptions and Contrasts 


An important assumption in analysis of variance is that the population variances are 
equal—that is, the groups have approximately the same spread. When variances differ 
markedly, a transformation may remedy the problem. For example, sometimes it helps 
to take the square root of each value of the outcome variable (or log transform each 
value) and use the transformed value in the analysis. 

In this example, we use a subset of the cases from the SURVEY? data file to address 
the question, “For males, does average income vary with education?" We focus on the 
following who: 


m Did not graduate from high school (HS dropout) 
W Graduated from high school (HS grad) 
W Attended some college (Some college) 
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m Graduated from college (College grad) 
m Have an M.A. or Ph.D. (Degree +) 


For each male subject (case) in the SURVEY? data file, use the variables /VCOME and 
EDUCS. The means, standard deviations, and sample sizes for the five groups are 
shown below: 


HS dropout HS grad Some college College grad Degree + 


mean $13,389 $21,231 $29,294 $30,937 $38,214 
sd 10,639 13,176 16,465 16,894 18,230 
n 18 39 17 16 14 


Visually, as you move across the groups, you see that average income increases. But 
considering the variability within each group, you might wonder if the differences are 
significant. Also, there is a relationship between the means and standard deviations— 
as the means increase, so do the standard deviations. They should be independent. 
Suppose you take the square root of each income value, there is less variability among 
the standard deviations, and the relation between the means and standard deviations is 


weaker: 


HS dropout HS grad Some college College grad Degree + 
mean 3.371 4.423 5.190 5.305 6.007 
sd 1.465 1.310 1.583 1.725 1.516 


A bar chart for the data will show the effect of the transformation. 


The input is: 


USE SURVEY2 

SELECT SEX$= 'Male' 

RECODE EDUCATN$-EDUCATN / 1,2='HS dropout', 3='HS grad', 
4='Some college', 5-'College, 
grad'6,7-'Degree +' 

CATEGORY EDUCATNS 

ORDER EDUCATNS / SORT = 'HS dropout' 'HS grad' 'Some college', 

'College grad' 'Degree +' 

BEGIN 

BAR INCOME * EDUCATN$ / SERROR FILL-.5 LOC--3IN,OIN 

BAR INCOME * EDUCATNS / SERROR FILL-.35 YPOW-.5, 
LOC-3IN,OIN 


END 
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The output is: 


INCOME 
INCOME 


0 0 
epee "MP 
Lt ty A ACA A 
EDUCATN$ EDUCATN$ 


In the chart on the left, you can see a relation between the height of the bars (means) 
and the length of the error bars (standard errors). The smaller means have shorter error 
bars than the larger means. After transformation, there is less difference in length 
among the error bars. The transformation aids in eliminating the dependency between 
the group and the standard deviation. 


To test for differences among the means: 


ANOVA 
LET SQRT_INC = SQR(INCOME) 
DEPEND SQRT_INC 
CATEGORY EDUCATNS 
ESTIMATE/NTEST = KS, SW, AD HTEST = LEVENE 


The output is: 


Dependent Variable | SQRT_INC 
N f 104 
Multiple R | 0.490801 
Squared Multiple R | 0.240886 


Analysis of Variance 


Source | Type III S df Mean Squares F-ratio p-value 
---------- hon ——Á— € Lo! ami! 
EDUCATNS | 4 17.155895 7.853793 0,000015 
Error 39 2.184409 
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Test for Homogeneity 


| Test Statistic 
Levene's Test 


Test for Normality 


0.079778 0.099782 
0.989641 0.608196 
0.424596 >0.15 


K-S Test (Lilliefors) 
Shapiro-Wilk Test 
Anderson-Darling Test 
From the above results of normality tests and the homogenity test, the assumption 
of normal residuals is satisfied and the transformed INCOME dependent variable 
fulfills the equal population variance assumption. 
The ANOVA table using the transformed income as the dependent variable suggests 
a significant difference among the four means (p-value < 0.0005). 


Tukey Pairwise Mean Comparisons 


Which means differ? This example uses the Tukey method to identify significant 
differences in pairs of means. Hopefully, you reach the same conclusions using either 
the Tukey or Bonferroni methods. However, when the number of comparisons is very 
large, the Tukey procedure may be more sensitive in detecting differences; when the 
number of comparisons is small, Bonferroni may be more sensitive. 


The input is: 


HYPOTHESIS 
POST EDUCATN$ / TUKEY 
TEST 


The output is: 


Post Hoc Test of SQRT_INC 
Using least squares means. 


Using model MSE of 2.184409 with 99 df. 
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Tukey's Honestly-Significant-Difference Test 


EDUCATNS (i) EDUCATNS (3) Difference p-value 95.0% Confidence Interval 


HS dropout HS grad -1.051736 0.099508 -2.221989 0.118517 
HS dropout Some college -1.819060 0.003917 -3.208003 -0.430118 
HS dropout College grad -1.934562 0.002209 -3.345650 -0.523474 
HS dropout Degree * -2.635771 0.000028 -4.099247 -1. 

HS grad Some college -0.767324 0.387284 -1.960895 0.42 

HS grad College grad -0.882826 0.267967 -2.102096 0.3 

HS grad Degree * -1.584035 0.007454 -2.863571 -0. E 
Some college College grad .115502 0.999430 -1.545987 1.314984 
Some college Degree + .816711 0.544944 -2.298899 0.665478 
College grad Degree + -0.701209 0.694029 -2.204170 0.801752 


The layout of the output panels for the Tukey method is the same as that for the 
Bonferroni method. Look first at the probabilities in the fourth column. Four of the 
probabilities indicate significant differences (they are less than 0.05). In the third 
column, row 2, 3 and 4, the average income for high school dropouts differs from those 
with some college (p-value — 0.003), from college graduates (p-value — 0.002), and 
also from those with advanced degrees (p-value < 0.0005). The seventh row shows that 
the differences between those with advanced degrees and the high school graduates are 
significant (p-value — 0.007). 


Contrasts 


In this example, the five groups are ordered by their level of education, so you use these 
coefficients to test linear and quadratic contrasts: 


Linear s2 47 "0 WI 2 
Quadratic 2-1 -2-1 2 


Then you ask, “Is there a linear increase in average income across the five ordered 
levels of education?" *A quadratic change?" 
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The input is: 


HYPOTHESIS 
NOTE 'Test of linear contrast', 
'across ordered group means' 
EFFECT EDUCATNS 
CONTRAST [-2 -1 O 1 2] 
TEST 


HYPOTHESIS 

NOTE 'Test of quadratic contrast', 
'across ordered group means' 

EFFECT EDUCATN$ 

CONTRAST [2 -1 -2 -1 2] 

TEST 

SELECT 


The output is: 


Test of linear contrast 
across ordered group means 


Test for effect called: EDUCATNS 


A Matrix 


0.000000 -4.000000 -3.000000  -2.000000 71.000000 


Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 

1 Upper 
— MÀ A mm ig uc 
A i 6.154368 1.141086 .182895 


Test of Hypothesis 
Source 1 ss df Mean Squares F-ratio p-value 


Hypothesis 63.542478 1 63.542478 — 29.089094 0.000000 
Error | 216.256486 99 2.184409 


Test of quadratic contrast 
across ordered group means 


Test for effect called: EDUCATNS 


A Matrix 


.000000 -3.000000 


0.000000 0.000000 -3.000000 e 


Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 
1 Lower Upper 


---------- + 
A i -1.352877 1.347611 -1.386568 -1.319187 
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Contrast Estimate 


Hypothesis | Estimate(AB) Standard Error 95.0% Confidence Interval 

! Lower Upper 
T Ó—P ————— A E O 
A i -1.352877 1.347611 -1.386568 -1.319187 


Test of Hypothesis 


Source ; SS df Mean Squares  F-ratio p-value 
Ne sare Sa MER DRE a rah a E EE 
Hypothesis | 2.201515 1 2.201515 1.007831 0.317870 
Error | 216.256486 99 2.184409 


The F-ratio for testing the linear contrast is 29.089 (p-value « 0.0005); for testing the 
quadratic contrast, it is 1.008 (p-value — 0.318). Thus, you can report that there is a 
highly significant linear increase in average income across the five levels of education 
and that you have not found a quadratic component in this increase. 


Example 3 
Two-Way ANOVA 


Consider the following two-way analysis of variance design from Afifi and Azen 
(1972), cited in Kutner (1974). The dependent variable, SYSINCR, is the change in 
systolic blood pressure after administering one of four different drugs to patients with 
one of three different diseases. Patients were assigned randomly to one of the possible 
drugs. The data are stored in the SYSTAT file AFIFI. 


To obtain a least-squares two-way analysis of variance, the input is: 


ANOVA 
USE AFIFI 
CATEGORY DRUG DISEASE 
DEPEND SYSINCR 
SAVE MYRESIDS / RESID DATA 
ESTIMATE 


Because this is a factorial design, ANOVA automatically generates an interaction term 
(DRUG * DISEASE). 
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The output is: 


Dependent Variable | SYSINCR 
N i 58 
Multiple R H 0.675 
Squared Multiple R | 0.456 


Analysis of Variance 
| Type III SS df Mean Squares F-ratio p-value 


| 2997.472 3 999.157 9.046 0.000 
DISEASE à 415.873 2 207.937 1.883 0.164 
DRUG*DISEASE | 707.266 6 117.878 1.067 0.396 
Error | ..5080.817 46 110.453 
Least Squares Means. Least Squares Means. 
32.0; 27.0 
26.2 23.6 
ba 20.2 
14.6| 16.8 
8.8 13.4 
SOE gtr YER o ie aay ORE: 
DRUG DISEASE 
Least Squares Means 
1 2 
4100 41.00, 
ant 2975 
jae jue 
-| 725 
-400' 400 
au $ 5 
1 PE. 4 1 ry 
3 
41.00, 
2975 
x 
aus 
725 
4.00 
1 2 3 4 
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In two-way ANOVA, begin by examining the interaction. If the interaction is 
significant, you must condition your conclusions about a given factor's effects on the 
level of the other factor. The DRUG * DISEASE interaction is not significant 
(p-value — 0.396), so shift your focus to the main effects. 

The DRUG effect is significant (p-value « 0.0005), but the DISEASE effect is not 
(p-value — 0.164). Thus, at least one of the drugs differs from the others with respect 
to blood pressure change, but blood pressure change does not vary significantly across 
diseases. 

For each factor, SYSTAT produces a plot of the average value of the dependent 
variable for each level of the factor. For the DRUG plot, drugs 1 and 2 yield similar 
average blood pressure changes. However, the average blood pressure change for 
drugs 3 and 4 are much lower. ANOVA tests for significant differences illustrated in 
this plot. 

For the DISEASE plot, we see a gradual decrease in blood pressure change across 
the three diseases. However, this effect is not significant; there is not enough variation 
among these means to overcome the variation due to individual differences. 

In addition to the plot for each factor, SYSTAT also produces plots of the average 
blood pressure change at each level of DRUG for each level of disease. Use these plots 
to illustrate interaction effects. Although the interaction effect is not significant in this 
example, we can still examine these plots. 

In general, we see a decline in blood pressure change across drugs. (Keep in mind 
that the drugs are only artificially ordered. We could reorder the drugs, and although 
the ANOVA results would not change, the plots would differ.) The similarity of the 
plots illustrates the nonsignificant interaction. 

A close correspondence exists between the factor plots and the interaction plots. The 
means plotted in the factor plot for D/SEASE correspond to the weighted average of 
the four points in each of the interaction plots. Similarly, each mean plotted in the 
DRUG factor plot corresponds to the weighted average of the three corresponding 
points across interaction plots. Consequently, the significant DRUG effect can be seen 
in the differing means in each interaction plot. Can you see the nonsignificant 
DISEASE effect in the interaction plots? 


Least-Squares ANOVA 


If you have an orthogonal design (equal number of cases in every cell), you will find 
that the ANOVA table is the same one you get with any standard program. SYSTAT 
can handle non-orthogonal designs, however (as in the present example). To 
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understand the sources for sum of squares, you must know something about least- 
squares ANOVA. 

As with one-way ANOVA, your specifying factor levels causes SYSTAT to create 
dummy variables out of the classifying input variable. SYSTAT creates one fewer 
dummy variable than the categories specified. 

Coding of the dummy variables is the classic analysis of variance parameterization, 
in which the sum of effects estimated for a classifying variable is 0 (Scheffé, 1959). In 
our example, DRUG has four categories; therefore, SYSTAT creates three dummy 
variables with the following scores for subjects at each level: 


1 0 0 for DRUG = 1 subject 

0 1 0 for DRUG - 2 subjects 

0 0 1 for DRUG = 3 subjects 

-1 -l -1 for DRUG = 4 subjects 

Because DISEASE has three categories, SYSTAT creates two dummy variables to be 
coded as follows: 


1 O for DISEASE = 1 subject 
0 | for DISEASE = 2 subjects 
-1 -1 for DISEASE = 3 subjects 


Now, because there are no continuous predictors in the model (unlike the analysis of 
covariance), you have a complete design matrix of dummy variables as follows 
(DRUG is labeled with an a, DISEASE with a b, and the grand mean with an m): 


Treatment Mean DRUG DISEASE Interaction 
A B m |al a2 a3| bl b2 |albl alb2 a2bl a2b2 a3bl a3b2 
1 1 dc 69 1 1 0 0 0 0 0 
1 (pO) PP d 0 1 0 1 0 0 0 0 
1 1 VIARIA S en -1 -1 0 0 0 0 
2 1 Y 0 1 1 0 0 1 0 0 0 
2 Moo Os eel d 0 1 0 0 0 1 0 0 
2 3 JA or E - 5 0 0 -l -1 0 0 
3 1 NL a 1 0 0 0 0 1 0 
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qs QUO ^. 0 1 0 0 
Y OF 09'.T- A 0 0 
Y =p 1 A 0 
Y Ai icc 0 1 0n 4 0 =l 0 -l 
E A er | 4 1 1 1 1 1 l 


Se kw UU 


This example is used to explain how SYSTAT gets an error term for the ANOVA table. 
Because it is a least-squares program, the error term is taken from the residual sum of 
squares in the regression onto the above dummy variables. For non-orthogonal 
designs, this choice is identical to that produced by GLM with Type III sum of squares. 
These, in general, will be the hypotheses you want to test on unbalanced experimental 
data. You can construct other types of sum of squares by using an A matrix or by 
running your ANOVA model using the Stepwise options in GLM. Consult the 
references and or Chapter 1: Linear Models of Statistics 1I if you do not already know 
what these sum of squares mean. 


Simple and deviation contrasts 


It is evident that only the main effect for DRUG is significant; therefore, you might 
want to test some specified contrasts on the DRUG effects. To compare a specified 
drug level with other drug levels, we can use the SIMPLE contrast and to compare each 
drug level with the mean of other DRUG levels, we can use DEVIATION contrast. 


The input is: 


PLENGTH LONG 
HYPOTHESIS 

EFFECT DRUG 

CONTRAST / DEVIATION [4] 
TEST 


HYPOTHESIS 

EFFECT DRUG 
CONTRAST / SIMPLE [4] 
TEST 
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The following are the results of the above hypothesis tests: 


Test for effect called: DRUG 


A Matrix 


Contrast Estimate 
Estimate (AB) Standard Error 95.0$ Confidence Interval 


Hypothesis | 
i Lower Upper 
o HE 
Al ! -9.380 3.202 -9.460 -9.300 
A2 H -10.128 3.202 -10.208 -10.048 
A3 i 12.287 3.474 12.200 12.374 


Test of Hypothesis 
ss df Mean Squares F-ratio p-value 


Source | 
-------- + 
Al t 948.050 1 948.050 8.583 0.005 
A2 | 1105.320 1 1105.320 10.007 0.003 
A3 | 1381.771 1 1381.771 12.510 0.001 
A | 2997.472 3 999.157 9.046 0.000 
Error | 5080.817 46 110.453 


Note that simultaneously and marginally each level of DRUG differs significantly from 
the mean of the other DRUG levels. 


Test for effect called: DRUG 


A Matrix 
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A Matrix 
i 11 12 
pee ee ao Sa 
1 } 0.000 0.000 
2 ; 0.000 0.000 
3 | 0.000 0.000 


Contrast Estimate 


Hypothesis | Estimate (AB) Standard Error 95.0% Confidence Interval 

i Lower Upper 
pC pu dus JO C ERI iie Re a E 6 Lope eet O 
Al i 12.450 3.811 12.355 12.545 
A2 i 13.011 3.811 12.916 13.106 
A3 i -3.800 4.070 -3.902 -3.698 


Test of Hypothesis 


Source | ss df Mean Squares F-ratio p-value 
See bmn nena a a a nnn nen nanan nen seee anne nen 
Al i 1178.892 1 1178.892 10.673 0.002 
A2 ! 1287.550 1 1287.550 11.657 0.001 
A3 i 96.267 2 96.267 0.872 0.355 
A | 2997.472 3 999.157 9.046 0.000 
Error | 5080.817 46 110.453 


Observe that A3 (p-value — 0.355397) is insignificant, that is, only the third and the 
fourth DRUG levels are not significantly different. 


Custom Contrasts 


A simple way to test DRUG contrasts would be to use the Bonferroni method to test 
all pairwise comparisons (Miller, 1985) of marginal drug means. However, to compare 
three or more means, you must specify the particular contrasts of interest. Here, we 


compare the first and third drugs, the first and fourth drugs, and the first two drugs with 
the last two drugs. 


The input is: 


HYPOTHESIS 

EFFECT DRUG 
CONTRAST [1 O -1 0] 
TEST 

HYPOTHESIS 

EFFECT DRUG 
CONTRAST [1 0 0 -1 
TEST 

HYPOTHESIS 

EFFECT DRUG 
CONTRAST [1 1 -1 -1] 
TEST 
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You need four numbers in each contrast because DRUG has four levels. You cannot use 
CONTRAST to specify coefficients for interaction terms. It creates an A matrix only for 
main effects. The following are the results of the above hypothesis tests: 
Test for effect called: DRUG 

A Matrix 


0.000 1.000 0,000  -1.000 0.000 


A Matrix 


0.000 0.000 0.000 0.000 


0.000 0.000 
Test of Hypothesis 


Source i SS df Mean Squares  F-ratio p-value 
Hypothesis | 1697.545 1 1697.545 15.369 0.000 
Error i 5080.817 46 110.453 


Test for effect called; DRUG 
A Matrix 


0.000 2.000 1.000 1.000 0.000 


A Matrix 


A Matrix 


0.000 0.000 
Test of Hypothesis 


Source i ss df Mean Squares F-ratio 
Miena ie A E 
Hypothesis | 1178.892 1 1178.892 10.673 
Error 1 5080.817 46 110.453 


Test for effect called: DRUG 
A Matrix 


0.000 2.000 2.000 0.000 0.000 
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0.000 0.000 0.000 
A Matrix 

0.000 0.000 

Test of Hypothesis 


Source i ss 


df Mean Squares F-ratio 


p-value 


ee mi ciim i ia A ae i ai mum 


Hypothesis | 2982.934 
Error | 5080.817 


1 2982.934 27.006 
46 110.453 


0.000 


Notice the A matrices in the output. SYSTAT automatically takes into account the 
degree of freedom lost in the design coding. Also, notice that you do not need to 
normalize contrasts or rows of the A matrix to unit vector length, as in some ANOVA 
programs. If you use (2 0 -2 0) or (0.707 0 -0.707 0) instead of (1 0 -1 0), you get the 


same sum of squares. 


For the comparison of the first and third drugs, the F-ratio is 15.369 
(p-value < 0.0005), indicating that these two drugs differ. Looking at the Quick Graphs 
produced earlier, we see that the change in blood pressure was much smaller for the 


third drug. 


Notice that in the A matrix created by the contrast of the first and fourth drugs, you 
get (2 1 1) in place of the three design variables corresponding to the appropriate 
columns of the A matrix. Because you selected the reduced form for coding of design 
variables in which sums of effects are 0, you have the following restriction for the 


DRUG effects: 


Q4 +A) + 03+ 04=0 


where each value is the effect for that level of DRUG. This means that: 


Q4 = -( A+ A+ 03) 


and the contrast DRUG(1) - DRUG(4) is equivalent to: 


a, -[- (a, + a+ 83 )] 2 0 


which is: 


20, + 0504-0 
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For the final contrast, SYSTAT transforms the (1 1 -1 -1) specification into contrast 
coefficients of (2 2 0) for the dummy coded variables. The p-value (< 0.0005) indicates 
that the first two drugs differ from the last two drugs. 


Simple Effects 


You can do simple contrasts between drugs within levels of disease (although the lack 
of a significant DRUG * DISEASE interaction does not justify it). To show how it is 
done, consider a contrast between the first and third levels of DRUG for the first 
DISEASE only. You must specify the contrast in terms of the cell means. Use the 
terminology: 

MEAN (DRUG index, DISEASE index) = M{i,j} 


You want to contrast cell means M (1,1) and M {3,1}. These are composed of: 
M{1,1} = pnta, * Bi au 
M(3,1) = p+ o+ Bi * as 
Therefore the difference between the two means is: 
M(1,1) - M(3,1) = a, — a + api - asi 


Now, suppose you consider the coding of the variables, you can construct an A matrix 
that picks up the appropriate columns of the design matrix. Here are the column labels 
ofthe design matrix (a means DRUG and b means DISEASE) to serve as a column ruler 
over the A matrix specified in the hypothesis. 


m| al a2 a3 bi b2 | albl alb2 a2bl a2b2 a3bl a3b2 
0 1 0 -l 0 0 1 0 0 0 -l 0 


The input is: 


HYPOTHESIS 
AMATRIX [0 10 -1001000 -1 0] 
TEST 
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The output is: 


A Matrix 


0.000 0.000 


-1.000 0.000 
Test of Hypothesis 
Source SS df Mean Squares F-ratio p-value 


Hypothesis 


338.000 1 338.000 3.060 0.087 
Error 


5080.817 46 110.453 


After you understand how SYSTAT codes design variables and how the model 
sentence orders them, you can take any standard ANOVA text like Winer, Brown and 
Michels (1991) or Scheffé (1959) and construct an A matrix for any linear contrast. 


Contrasting Marginal and Cell Means 


Now look at how to contrast cell means directly without being concerned about how 


they are coded. Test the first level of DRUG against the third (contrasting the marginal 
means). 


The input is: 


HYPOTHESIS 
SPECIFY DRUG[1] = DRUG[3] 
TEST 


To contrast the first against the fourth: 
HYPOTHESIS 


SPECIFY DRUG[1] = DRUG[4] 
TEST 
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Finally, here is the simple contrast of the first and third levels of DRUG for the first 
DISEASE only: 


HYPOTHESIS 
SPECIFY DRUG[1] DISEASE[1] = DRUG[3] DISEASE[1] 
TEST 


Screening Results 


Let's examine the AFIFI data in more detail. To analyze the residuals to examine the 
ANOVA assumptions, first plot the residuals against estimated values (cell means) to 
check for homogeneity of variance. Use the Studentized residuals to reference them 
against a t distribution. In addition, stem-and-leaf plots of the residuals and boxplots of 
the dependent variable aid in identifying outliers. 


The input is: 


ANOVA 
USE AFIFI 
CATEGORY DRUG DISEASE 
DEPEND SYSINCR 
SAVE MYRESIDS / RESID DATA 
ESTIMATE 
DENSITY SYSINCR * DRUG / BOX 
USE MYRESIDS 
PLOT STUDENT*ESTIMATE / SYM=1 FILL=1 
STEM STUDENT 


50, T T T -——— 2 =F T T 
| EA E 
a i ea wen 
| : ea 
g (Ex e fer Do: 
$ a AO cr 
Z 20 18 Sada - 
[2 BAF - » 
10 ] NET T 
0 " WE 
c = AL 2 A mb ol 
QUEE MEC E 0 $) 10 20 30 40 
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Dependent Variable | SYSINCR 


N 58 

Multiple R i 0.675 

Squared Multiple R | 0.456 

Analysis of Variance 

Source | Type III SS df Mean Squares F-ratio p-value 
EEE E LA A EA e ames ania ere 
DRUG H 2997.472 3 999.157 9.046 0.000 
DISEASE i 415.873 2 207.937 1.883 0.164 
DRUG*DISEASE | 707.266 6 117.878 1.067 0.396 
Error t 5080.817 46 110.453 


The plots suggest the presence of an outlier. The smallest value in the stem-and-leaf 
plot seems to be out of line. A £ statistic value of -2.647 corresponds to p-value < 0.01, 
and you would not expect a value this small to show up in a sample of only 58 
independent values. In the scatterplot, the point corresponding to this value appears at 
the bottom and badly skews the data in its cell (which happens to be DRUG/, 
DISEASE3). The outlier in the first group also clearly stands out in the boxplot. To see 
the effect of this outlier, delete the observation with the outlying Studentized residual. 


Then, run the analysis again: 
Stem and Leaf Plot of Variable: STUDENT, N = 58 


Minimum 1? 72.647 

Lower Hinge : -0.761 

Median : 0.101 

Upper Hinge : 0.698 

Maximum : 1.552 
“2 6 

=2 

-1 987666 

1 410 

-0 H 9877765 

-0 4322220000 

0 M 001222333444 

0 H 55666888 

1 

1 


011133444 
55 


The differences are not substantial. Nevertheless, notice that the DISEASE effect is 
substantially attenuated when only one case out of 58 is deleted. Daniel (1960) gives 
an example in which one outlying case alters the fundamental conclusions of a 
designed experiment. The F-test is robust to certain violations of assumptions, but 
factorial ANOVA is not robust against outliers. You should routinely do these plots for 
ANOVA. 
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Example 4 
Pairwise comparisons 


An analysis of variance indicates whether (at least) one of the groups differs from the 
others. However, you cannot determine which group(s) differ(s) based on ANOVA 
results. To examine specific group differences, use post hoc tests. 

In this example, we use the AFIFI data to test for the difference between DRUG 
levels. 


The input is: 


ANOVA 
USE AFIFI 
DEPEND SYSINCR 
CATEGORY DRUG 
ESTIMATE/HTEST = LEVENE 


The output is: 
Dependent Variable | SYSINCR 
N 1 58 
Multiple R 1 0.579 
Squared Multiple R | 0.335 


Analysis of Variance 
Source | Type III SS df Mean Squares F-ratio p-value 


DRUG 1 3133.239 3 1044.413 9.086 0.000 
Error | 6206.917 54 114.943 


Test for Homogeneity 


| Test Statistic p-value 
necem doe Lem eee wenn nanan nnn == 
Levene's Test | 0.246 0.864 


In the output, by looking at the test for homogeneity (Levene's test statistics — 0.246, 
p-value — 0.864), you conclude that the dependent variable SYSINCR fulfills the equal 
population variance assumption. 

In the ANOVA table, the p-value (<0.0005), indicates that the null hypothesis of 
equal means is overwhelmingly rejected. The F-test in an analysis of variance only 
indicates that not all group means are equal. However, one may be interested in 
determining the group(s) that differ(s), that is, the groups that are responsible for the 
rejection of the null hypothesis of equal means. To examine specific group differences 
and perhaps to order the groups according to their means, one may use the following 


post hoc tests. 
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The input is: 
HYPOTHESIS 
POST DRUG / SNK 
TEST/CONFI=0.95 
The output is: 


Post Hoc Test of SYSINCR 
Using least squares means. 
Using model MSE of 114.943 with 54 df. 


Student-Newman-Keuls Test 


SubGroup DRUG 


1 3 . . 

4 13.500 15.000 0,241 
2 2 25.533 12.000 

1 26.067 16.000 0.895 


* This test controls family-wise error rate under the complete null hypothesis but not 
under partíal null hypothesis. 


The Student-Newman-K euls test displays homogeneous subset numbers, factor levels, 
ordered group means, group size, and p-value for each subset of the treatments under 
consideration. The above output shows that groups 3 and 4 belong to the same 
homogeneous subset (the corresponding p-value is 0.241), whereas the rest of the 
groups belong to another subset (p-value is 0.895). 


Example 5 
Unbalanced ANOVA 


To test the effect of DRUG, DISEASE, and DRUG * DISEASE interaction on the 
response variable, three different types of sum of squares are used. 


The input is: 


ANOVA 
USE AFIFI 
DEPEND SYSINCR 
CATEGORY DRUG DISEASE 
ESTIMATE/SS = TYPE1 
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ANOVA 
DEPEND SYSINCR 
CATEGORY DRUG DISEASE 
ESTIMATE/SS = TYPE2 


ANOVA 
DEPEND SYSINCR 
CATEGORY DRUG DISEASE 
ESTIMATE/SS = TYPE3 


The output is: 
Dependent Variable | SYSINCR 
N | 58 
Multiple R | 0.6175 
Squared Multiple R | 0.456 


Analysis of Variance 


Source j F-ratio p-value 
esa ari n i db i 

DRUG 1| 3133.239 3 

DISEASE i 418.834 2 209.417 1.896 0.162 

DRUG*DISEASE | 707.266 6 117.878 1.067 0.396 

Error i 5080.817 46 110.453 


Analysis of Variance 


II SS df Mean Squares F-ratio p-value 


DRUG 3.433 3 1021.144 9.245 0.000 
DISEASE 418.834 2 209.417 1.896 0.162 
DRUG*DISEASE 707.266 6 117.878 1.067 0.396 
Error 5080.817 46 110.453 


Analysis of Variance 


Source | Type III SS df Mean Squares 


DRUG f 2997.472 3 999.157 
DISEASE f 415.873 2 207.937 
DRUG*DISEASE | 707.266 6 117.878 
Error i 5080.817 46 110.453 


Note the differences between the three types of sum of squares. The Type I sum of 
squares for DRUG essentially tests the differences between the expected values of the 
arithmetic mean response for different drugs; testing the effect of the disease is not 
taken into account. The Type II sum of squares for DRUG measures the difference 
between the arithmetic means for each drug after adjusting for the disease. The Type 
III sum of squares measures the difference between the least-squares means for drug 
levels. 

No matter which sum of squares you use, the above analysis shows significant 
differences among the four drugs, while the DISEASE effect and the DRUG *DISEASE 


interaction are not significant. 
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Example 6 
Single-Degree-of-Freedom Designs 


The data in the REACT file involve yields of a chemical reaction under various 
combinations of four binary factors (A, B, C, and D). Two reactions were observed 
under each combination of experimental factors, so the number of cases per cell is two. 
To analyze the data in a four-way ANOVA, the input is: 
ANOVA 
USE REACT 
CATEGORY A, B, C, D 


DEPEND YIELD 
ESTIMATE 


You can see the advantage of ANOVA over GLM when you have several factors; you 
have to select only the main effects. With GLM, you have to specify the interactions 
and identify which variables are categorical (that is, 4, B, C, and D). The following 
example is the full model using GLM: 

MODEL YIELD = CONSTANT + A+ B «4 C « D +, 


A*B + A*C + A*D + B*C + B*D + C*D +, 
A*B*C + A*B*D + A*C*D + B*C*D +, 


A*B*C*D 
The output is: 
Dependent Variable | YIELD 
N t 32 
Multiple R | 0.755 
Squared Multiple R | 0.570 


Analysis of Variance 


Source | Type III SS df Mean Squares 
--------- * 

A i 369800.000 1 369800.000 
B i 1458.000 1 1458.000 
c H 5565.125 1 5565.125 
D | 172578.125 1 172578.125 
A*B H 87153.125 1 87153.125 
A*C i 137288.000 1 137288.000 
A*D i 328860.500 1 328860.500 
B*C i 61952.000 1 61952.000 
B*D i 3200.000 H 3200.000 
C*D H 3160.125 1 3160.125 
A*B*C t 81810.125 1 81810.125 
A*B*D H 4753.125 1 4753.125 
A*C*D i 415872.000 1 415872.000 
B*C*D i 4.500 1 4.500 
A*B*C*D | 15051.125 1 15051.125 
Error ! 1272247.000 16 79515.438 


The output shows a significant main effect for the first factor (A) plus one significant 
interaction (4*C*D). 
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Assessing Normality 


Let’s look at the study more closely. Because this is a single degree of freedom study 
(a 2" factorial), each effect estimate is normally distributed if the usual assumptions for 
the experiment are valid. All of the effects estimates, except the constant, have zero 
mean and common variance (because dummy variables were used in their 
computation). Thus, you can compare them to a normal distribution. SYSTAT 
remembers your last selections. 


The input is: 


SAVE EFFECTS / COEF 
ESTIMATE 


This reestimates the model and saves the regression coefficients (effects). The file has 
one case with 16 variables (CONSTANT plus 15 effects). The effects are labeled X(1), 
X(2), and so on because they are related to the dummy variables, not the original 
variables A, B, C, and D. Let's transpose this file into a new file containing only the 15 
effects and create a probability plot of the effects. 


The input is: 


USE EFFECTS 

DROP CONSTANT 

TRANSPOSE 

PPLOT col(1) / FILL=1 SYMBOL=1, 
XLABEL="Estimates of Effects” 
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The output is: 
2 
2 
$1 
o 
3 
6 
a 
T0 
E 
o 
z 
ET “100 0 100 200 
Estimates of Effects 


These effects are indistinguishable from a random normal variable. They plot almost 
on a straight line. What does it mean for the study and for the significant F-test? 


It is time to reveal that the data were produced by a random number generator. 


If you are doing a factorial analysis of variance, the p-value you see on the output 
are not adjusted for the number of factors. If you do a three-way design, look at 
seven tests (excluding the constant). For a four-way design, examine 15 tests. Out 
of 15 F-test on random data, expect to find at least one test approaching 
significance. You have two significant and one almost significant, which is not far 
out of line. The probabilities for each separate F-test need to be corrected for the 
experimentwise error rate. Some authors devote entire chapters to fine distinctions 
between multiple comparison procedures and then illustrate them within a 
multifactorial design not corrected for the experimentwise error rate just 
demonstrated. Remember that a factorial design is a multiple comparison. If you 
have a single-degree-of-freedom study, use the procedure you used to draw a 


probability plot of the effects. Any effect that is really significant will become 
obvious. 


If you have a factorial study with more degrees of freedom on some factors, use the 
Bonferroni critical value for deciding which effects are significant. It guarantees 
that the Type I error rate for the study will be no greater than the level you choose. 
In the above example, this value is 0.05 / 15 (that is, 0.003). 


Multiple F-tests based on a common denominator (mean-square error in this 
example) are correlated. This complicates the problem further. In general, the 
greater the discrepancy between numerator and denominator degrees of freedom 


II-151 
Linear Models II: Analysis of Variance 


and the smaller the denominator degrees of freedom, the greater the dependence of 
the tests. The Bonferroni tests are best in this situation, although Feingold and 
Korsog (1986) offer some useful alternatives. 


Example 7 
Separate Variance Hypothesis Tests 


The data in the MJ20 data file are from Milliken and Johnson (1984). They are the 
results of a paired-associate learning task. GROUP describes the type of drug 
administered; LEARNING is the amount of material learned during testing. First we 
perform Levene's test (Levene, 1960) to determine if the variances are equal across 
cells. 


The input is: 


ANOVA 
USE MJ20 
SAVE MJRESIDS / RESID DATA 
DEPEND LEARNING 
CATEGORY GROUP 
ESTIMATE 
USE MJRESIDS 
LET RESIDUAL - ABS (RESIDUAL) 
CATEGORY GROUP 
DEPEND RESIDUAL 
ESTIMATE 


The following is the ANOVA table of the absolute residuals: 


Dependent Variable ; RESIDUAL 
N i 29 
Multiple R f 0.675 
Squared Multiple R | 0.455 


Analysis of Variance 
Source | Type III SS df Mean Squ 


GROUP | 30.603 3 . 
Error } 36.608 25 1.464 


Notice that the F-ratio is significant, indicating that the separate variances test is 
advisable. Let us do several single-degree-of-freedom tests, following Milliken and 


Johnson. The first is for comparing all drugs against the control; the second tests the 
hypothesis that groups 2 and 3 together are not significantly different from group 4. 
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The input is: 
ANOVA 
USE MJ20 
CATEGORY GROUP 
DEPEND LEARNING 
ESTIMATE 
HYPOTHESIS 


SPECIFY 3*GROUP[1] = GROUP[2] «GROUP[3] + GROUP[4] / SEPARATE 


TEST 
HYPOTHESIS 


SPECIFY 2*GROUP[4] - GROUP[2] «GROUP[3] / SEPARATE 


TEST 


The ANOVA table has been omitted because it is not valid when variances are unequal. 


The output is: 
Using separate variances estimate for error term. 
A Matrix 
1 2 3 4 


Null Hypothesis Value for D 
0.000 

Null Hypothesis Contrast AB-D 
-20.327 

Contrast Estimate 


Hypothesis | Estimate (AB-D) Standard Error 


A i 


Test of Hypothesis 


Source i 
----------- + 
Hypothesis | 242.720 1 242.720 
Error | 95.085 7.096 13.399 


Using separate variances estimate for error term. 


A Matrix 


Null Hypothesis Value for D 
0.000 

Null Hypothesis Contrast AB-D 
7.208 


95.0% Confidence Interval 
Lower Upper 
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Contrast Estimate 


Hypothesis | Estimate(AB-D) Standard Error 95.0% Confidence Interval 
1 L Upper 


7.250 


ss df Mean Squares  F-ratio p-value 
Hypothesis | 65.634 1 65.634 18.431 0.000 
Error | 72.452 20.346 3.561 


Example 8 
Analysis of Covariance 


Winer, Brown and Michels (1991) uses the COVAR data file for an analysis of 
covariance in which X is the covariate and TREAT is the treatment. Cases do not need 
to be ordered by the grouping variable TREAT. 

Before analyzing the data with an analysis of covariance model, be sure there is no 
significant interaction between the covariate and the treatment. The assumption of no 
interaction is often called the homogeneity of slopes assumption because it is 
tantamount to saying that the slope of the regression line of the dependent variable onto 
the covariate should be the same in all cells of the design. 

Parallelism is easy to test with a preliminary model. Use GLM to estimate this 
model with the interaction between treatment (TREAT) and covariate (X) in the model. 


The input is: 


GLM 
USE COVAR 


CATEGORY TREAT 
MODEL Y = CONSTANT + TREAT + X + TREAT*X 


ESTIMATE 
The output is: 
Dependent Variable | Y 
N Tot 
Multiple R | 0.921 


Squared Multiple R | 0.849 


Analysis of Variance 
n Squares F-ratio p-value 


Source | Type III SS df Mean Sq o oros 
TREAT a Le ES! 2 2: 
1 1 Mee 1 0.000 
TREAT*X | 0.667 2 Q 005 
Error | 9.635 15 
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The probability value for the treatment by covariate interaction is 0.605, so the 
assumption of homogeneity of slopes is plausible. 


Now, fit the usual analysis of covariance model by specifying: 


ANOVA 
USE COVAR 
PLENGTH MEDIUM 
CATEGORY TREAT 
DEPEND Y 
COVARIATE X 
ESTIMATE 


For incomplete factorials and similar designs, you still must specify a model (using 
GLM) to do analysis of covariance. 


The output is: 
Dependent Variable 
N 


Multiple R | 0.916 
Squared Multiple R ! 


Analysis of Variance 


Source | Type III SS df Mean Squares  F-ratio p-value 
ad A a A se em 
TREAT | 16.932 2 8.466 13.970 0.000 
x i 16.555 1 16.555 27.319 0.000 
Error | 10.302 17 0.606 


Least Squares Means 


Factor | Level LS M 
-------- + 
TREAT | 
TREAT j| 
TREAT | 


0.294 7.000 


* Means are computed after adjusting covariate effect. 


The treatment adjusted for the covariate is significant. There is a significant difference 
among the three treatment groups. Also, notice that the coefficient for the covariate is 
significant (F-ratio = 27.319, p-value « 0.0005). If it were not, the analysis of 
covariance could be taking away a degree of freedom without reducing mean-square 
error enough to help you. 

SYSTAT computes the adjusted cell means the same way it computes estimates 
when saving residuals. Model terms (main effects and interactions) that do not contain 
categorical variables (covariates) are incorporated into the equation by adding the 
product of the coefficient and the mean of the term for computing estimates. The grand 
mean (CONSTANT) is included in computing the estimates. 
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Example 9 
One-Way Repeated Measures 


In this example, six rats were weighed at the end of each of five weeks. A plot of each 
rat's weight over the duration of the experiment is shown below: 
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ANOVA is the simplest way to analyze this one-way model. Because we have no 
categorical variable(s), SYSTAT generates only the constant (grand mean) in the 
model. To obtain individual single-degree-of-freedom orthogonal polynomials, the 
input is: 

ANOVA 


USE RATS n 
DEPEND WEIGHT(1..5) / REPEAT NAME-"Time 


PLENGTH MEDIUM 
ESTIMATE 


The output is: 
N of Cases Processed : 6 


Dependent Variable Means 
WEIGHT (1) WEIGHT (2) WEIGHT (3) WEIGHT (4) WEIGHT (5) 


Univariate and Multivariate Repeated Measures Analysis 
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Within Subjects 


Sou: $ ss df Mean Squares F-ratio p-value 
Time | 134.4 33.617 16.033 0.000 
Error | 41.933 20 2.097 

Greenhouse-Geisser Epsilon | 0.342 

Huynh-Feldt Epsilon 1:0.427 


Single Degree of Freedom Polynomial Contrasts 


Polynomial Test of Order 1 (Linear 


Source F-ratio p-value 
Time | 114.817 1 . 38.572 0.002 
Error | 14.883 5 2.977 


Polynomial Test of Order 2 (Quadratic) 


Source 


| 18.107 1 18.107 7.061 
Error | 12.821 5 2.564 


Source | SS df Mean Squares F-ratio p-value 

Cp A A cx acini A ee a tnm 
Time + 1.350 1 1.350 0.678 0.448 
Error ; 9.950 5 1.990 


Source | ss df Mean Squares F-ratio p-value 
Be Se RES RE St a epee te A LS ere 
Time t 0.193 1 0.193 0.225 0.655 
Error | 4.279 5 0.856 


Multivariate Repeated Measures Analysis 


Test of: Time 


Statistic 1 

o-++-------------------- 4e-------------------------------- 
Wilks's Lambda i 0.011 4 

Pillai Trace | 0.989 4 
Hotelling-Lawley Trace | 86.014 4 


Value Hypothesis df Error df  F-ratio p-value 


2 43.007 0.023 
2 43.007 0.023 
2 43.007 0.023 


The Huynh-Feldt p-value (0.002) does not differ from the p-value for the F-ratio to any 
significant degree. Compound symmetry appears to be satisfied and weight changes 


significantly over the five trials. 


The polynomial tests indicate that most of the trials effect can be accounted for by 
a linear trend across time. In fact, the sum of squares for TIME is 134.467, and the sum 
of squares for the linear trend is almost as large (114.817). Thus, the linear polynomial 
accounts for roughly 85% of the change across the repeated measures. 
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Unevenly Spaced Polynomials 


Sometimes the underlying metric of the profiles is not evenly spaced. Let's assume that 
the fifth weight was measured after the tenth week instead of the fifth. In that case, the 
default polynomials have to be adjusted for the uneven spacing. These adjustments do 
not affect the overall repeated measures tests of each effect (univariate or multivariate), 
but they partition the sum of squares differently for the single-degree-of-freedom tests. 


The input is: 


ANOVA 
USE RATS 
DEPEND WEIGHT(1.. 5) / REPEAT-5(1 2 3 4 10) NAME-"Time" 


PLENGTH MEDIUM 
ESTIMATE 


Alternatively, you could request a hypothesis test, specifying the metric for the 
polynomials: 
HYPOTHESIS 


WITHIN 'Time' 
CONTRAST / POLYNOMIAL METRIC-1, 2,3,4,10 


TEST 


The last point has been spread out further to the right. 


The output is: 


Univariate and Multivariate Repeated Measures Analysis 


Within Subjects 


Mean Squares F-ratio 


Source | ss 
33.617 
20 2.097 
Greenhouse-Geisser Epsilon 1 0.342 
Huynh-Feldt Epsilon 1 0.427 
ingle Degree of Freedom Polynomial Contrasts 


Polynomial Test of Order l (Linear) 


Source | Ss df Squares — F-ratio p-value 
Beni pS. M EE ------------- 
Time ; 67.213 1 0.004 
Error | 14.027 5 


Polynomial Test of Order 2 (Quadratic) 


df Mean Squares 
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The significance tests for the linear and quadratic trends differ from those for the 
evenly spaced polynomials. Before, the linear trend was strongest; now, the quadratic 
polynomial has the most significant results (F-ratio = 107.9, p-value < 0.0005). 

You may have noticed that although the univariate F-tests for the polynomials are 
different, the multivariate test is unchanged. The latter measures variation across all 
components. The ANOVA table for the combined components is not affected by the 
metric of the polynomials. 

Difference Contrasts 
If you do not want to use polynomials, you can specify a C matrix that contrasts 
adjacent weeks. After estimating the model, use the following input: 
HYPOTHESIS 
WITHIN 'Time' 
CONTRAST / ADJDIFF 
TEST 
The output is: 
Multivariate Repeated Measures Analysis 
Test of: Time 
Statistic | Value Hypothesis df Error df  F-ratio p-value 
A5 usce ee NN Led resists iussus op ees secs mS iai ici del ide 
Wilks's Lambda i 0.011 4 2 43.007 0.023 

Pillai Trace | 0.989 4 2 43.007 0.023 

Hotelling-Lawley Trace | 86.014 4 2 43.007 0.023 
Notice the C matrix that this command generates. In this case, each of the univariate 
F-tests covers the significance of the difference between the adjacent weeks indexed 
by the C matrix. For example, the F-ratio = 17.241 shows that the first and second 
weeks differ significantly. The third and fourth weeks do not differ (F-ratio = 0.566). 
Unlike polynomials, these contrasts are not orthogonal. 

Summing Effects 


To sum across weeks, the input is: 


HYPOTHESIS 
WITHIN 'Time' 
CONTRAST / SUM 
TEST 
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The output is: 


C Matrix 


p 

i . 1 . 
Error | 19.333 5 . 
2 i 10.667 1 . 
Error | 1.333 5 . 
3 H 4.167 1 4. 
Error | 36.833 5 7. 
4 1 0.667 1 0. 
Error | 1.333 5 0.267 
Multivariate Test Statistics 

istic -ratio df p-value 

Wilks's Lambda | 0.01 43.007 4, 2 0.023 
Pillai Trace i 0.989 43.007 4,2 0.023 


Hotelling-Lawley Trace | 86.014 43.007 4,2 


In this example, you are testing whether the overall weight (across weeks) significantly 
differs from 0. Naturally, the F-ratio is significant. Notice the C matrix that is 
generated. It is simply a set of 1’s that, in the equation BC' - 0, sum all the coefficients 
in B. In a group-by-trials design, this C matrix is useful for pooling trials and analyzing 
group effects. 


Custom Contrasts 


To test any arbitrary contrast effects between dependent variables, you can use the C 
matrix, which has the same form (without a column for the CONSTANT) as the A 


matrix. The following commands test à linear trend across the five trials: 
HYPOTHESIS 


CMATRIX [-2 -1 0 1 2] 
TEST 
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The output is: 
C Matrix 
1 2 3 4 5 
1.000 1.000 1.000 1.000 1.000 
Test of Hypothesis 
Source i SS df Mean Squares F-ratio p-value 
me We A Pro e Macc lari ke pte icut cine OMEN TU agmen 
Hypothesis ; 6080.167 1 6080.167 295,632 0.000 
Error + 102.833 5 20.567 
C Matrix 
Test of Hypothesis 
1 . 1148.167 38.572 
Error i 148.833 29.767 
Example 10 


Repeated Measures ANOVA for One Grouping Factor and One Within Fac- 
tor with Ordered Levels 


The following example uses estimates of population for 1983, 1986, and 1990 and 
projections for 2020 for 57 countries from the OURWORLD data file. The data are log, 
transformed before analysis. Here you compare trends in population growth for 
European and Islamic countries. The variable GROUPS contains codes for these 
groups plus a third code for New World countries (we exclude these countries from this 
analysis). To create a bar chart of the data after using YLOG to log transform them: 


USE OURWORLD 

SELECT GROUP$<> 'NewWorld' 

BAR pop 1983.. pop 2020 / REPEAT OVERLAY YLOG, 
GROUP=group$SERROR FILL-.35,.8 
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To perform a repeated measures analysis: 


ANOVA 
USE OURWORLD 
SELECT GROUP$<> 'NewWorld' 
CATEGORY GROUP$ 
LET(POP_1983, POP_1986, POP_19 
DEPEND POP 1983 POP 1986 POP 1 
PLENGTH MEDIUM 
ESTIMATE 


The output is: 


Linear Models II: Analysis of Variance 


90, POP 2020) - L10 (@) 
990 POP_2020 / REPEAT-4 NAME-'Time' 


Univariate and Multivariate Repeated Measures Analysis 


Between Subjects 


Source | ss df 

eneke dr pe 
GROUPS ; 0.233 1 
Error | 30.794 34 


Within Subjects 


Source 


Time 
Time*GROUPS ; 
Error H 


| 0.528 


Greenhouse-Geisser Epsilon 
1 0.566 


Huynh-Feldt Epsilon 


p-value 


0.616 


Single Degree of Freedom Polynomial Contrasts 


Polynomial Test of Order 1 (Linear) 


Source df Mean Squares 
Time 1 0.675 
Time*GROUPS | 1 0.583 
Error 34 0.002 


F-ratio p-value G-G H-F 
7235.533 0.000 0.000 0.000 
208.352 0.000 0.000 0.000 
F-ratio p-value 
370.761 0.000 
320.488 0.000 
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Polynomial Test of Order 2 (Quadratic) 


Source 


Time 
Time*GROUPS | 
Error H 


Polynomial Test of Order 3 (Cubic) 


Source 

“Time 96.008 
Time*GROUPS . 1 0.027 94.828 
Error 34 0.000 


Multivariate Repeated Measures Analysis 


Test of: Time 


Statistic Hypothesis df Error 


157.665 
3 32 157.665 0.000 
3 32 157.665 0.000 


Wilks's Lambda 
Pillai Trace 
Hotelling-Lawley Trace 


Test of: Time*GROUP$ 


Hypothesis df Error df F-ratio p-value 


Statistic 


Wilks's Lambda 3 32 130.336 0.000 
Pillai Trace 3 32 130.336 0.000 
Hotelling-Lawley Trace 3 32 130.336 0.000 


The within-subjects results indicate highly significant linear, quadratic, and cubic 
changes across time. The pattern of change across time for the two groups also differs 
significantly (that is, the TIME * GROUPS interactions are highly significant for all 
three tests). 

Notice that there is a larger gap in time between 1990 and 2020 than between the 
other values. Let's incorporate "real time" in the analysis with the following 
specification: 

DEPEND POP 1983 POP 1986 POP 1990 POP 2020/REPEAT-4 (83,86,90,120), 


NAME= ‘TIME’ 
ESTIMATE 


The results for the orthogonal polynomials are shown below: 
Single Degree of Freedom Polynomial Contrasts 
Polynomial Test of Order 1 (Linear) 


Source I SS df Mean Squares F-ratio p-value 

E —€——————— P(— lilimtÁ! 
TIME | 0.831 1 0.831 317.273 0.000 
TIME*GROUPS | 0.737 1 0.737 281.304 0.000 
Error | 0.089 34 0.003 
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Polynomial Test of Order 2 (Quadratic) 


.003 1 
.001 1 0.001 


0.025 34 0.001 


of Order 3 (Cubic) 


| 0.000 
TIME*GROUPS | 0.000 1 
Error | 0.006 34 0.000 


When the values for POP. 2020 are positioned on a real time line, the tests for 
gnificant. The test for the linear 


quadratic and cubic polynomials are no longer si 
TIME * GROUPS interaction, however, remains highly significant, indicating that the 
slope across time for the Islamic group is significantly steeper than that for the 


European countries. 


Example 11 
Repeated Measures ANOVA for Two Grouping Factors and 


One Within Factor 


Repeated measures enables you to handle grouping factors automatically. The 


following example is from Winer, Brown and Michels (1991). There are two grouping 
factors (ANXIETY and TENSION) and one trial factor in the file REPEATI. The 
following is a dot display of the average responses across trials for each of the four 


combinations of ANXIETY and TENSION. 
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The input is: 
ANOVA 
USE REPEAT1 


LET TENS - TENSION 


DOT TRIAL(1..4) / Group=anxiety,tens, LINE, REPEAT, SERROR 
CATEGORY ANXIETY TENSION 


DEPEND TRIAL(1 .. 4) / REPEAT NAME-'Trial' 
PLENGTH MEDIUM 
ESTIMATE 


The model also includes an interaction between the grouping factors 
(ANXIETY * TENSION). 
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The output is: 
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Univariate and Multivariate Repeated Measures Analysis 


Between Subjects 


Source 


ANXIETY | 10.083 1 
TENSION p 8.333 1 
ANXIETY*TENSION | 80.083 1 

Error | 82.500 8 

Within Subjects 

Source i ss df 
Trial 991.500 3 
Trial*ANXIETY i 8.417 3 
Trial*TENSION | 12.167 3 
Trial*ANXIETY*TENSION | 12.750 3 
Error | 52.167 24 


Greenhouse-Geisser Epsilon | 0.536 
Huynh-Feldt Epsilon | 0.902 


10.083 

8.333 
80.083 
10.313 


Mean Squares F-ratio p-value G-G H-F 


330.500 152.051 0.000 0.000 0.000 
2.806 1.291 0.300 0.300 0.301 
4.056 1.866 0.162 0.197 0.169 
4.250 1.955 0.148 0.185 0.155 
2.174 


Single Degree of Freedom Polynomial Contrasts 


Polynomial Test of Order 1 (Linear) 


Source i 

Eataa ee al ls at n + 
Trial | 984.150 1 
Trial*ANXIETY i 1.667 1 
Trial*TENSION 110.417 1 
Trial*ANXIETY*TENSION | 9.600 1 
Error U^ 31.767 B 


Polynomial Test of Order 2 (Quadratic) 


Source i ss df 
Trial i 6.750 1 
Trial*ANXIETY | 3.000 1 
Trial*TENSION i 0.083 1 
Trial*ANXIETY*TENSION ¡ 0.333 1 
Error 1 15.833 8 


Polynomial Test of Order 3 (Cubic) 


Source 1 ss df 
Trial i 0.600 1 
Trial*ANXIETY | 3.750 1 
Trial*TENSION | 1.667 1 
Trial*ANXIETY*TENSION | 2.817 1 
Error | 4.567 8 


Multivariate Repeated Measures Analysis 
Test of: Trial 


Statistic 
Wilks's Lambda i 
Pillai Trace 1 0.985 
Hotelling-Lawley Trace | 63.843 


Mean S: 


984.150 247.845 0.000 
1.667 0.420 0.535 
10.417 2.623 0.144 
9.600 2.418 0.159 
3.971 


Mean Squares F-ratio p-value 


6.750 3,411 0.102 
3.000 1.516 0.253 
0.083 0.042 0.843 
0.333 0.168 0.692 
1.979 


3.750 6.569 0.033 
1.667 2.920 0.126 
2.817 4.934 0.057 
0.571 


Hypothesis df Error df F-ratio p-value 


3 6 127.686 0.000 
3 6 127.686 0,000 
3 6 127.686 0.000 
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Test of: Trial*ANXIETY 


Statistic | Value Hypothesis df Error df  F-ratio p-value 
Babe NK — lom cor [(Xcicoiocm 411022 goat Bie ronda so ai 
Wilks's Lambda | 0.244 3 6 6.183 0.029 
Pillai Trace i 0.756 3 6 6.183 0.029 
Hotelling-Lawley Trace | 3.091 3 6 6.183 0.029 
Test of: Trial*TENSION 

Statistic i Value Hypothesis df Error df  F-ratio p-value 
Wilks's Lambda | 0.361 3 6 3.546 0.088 
Pillai Trace | 0.639 3 6 3.546 0.088 
Hotelling-Lawley Trace | 1.773 3 6 3.546 0.088 
Test of: Trial*ANXIETY*TENSION 

Statistic i Value Hypothesis df Error df  F-ratio p-value 
Bori merni o MA co a eee remus o. cU ose fe alba 
Wilks's Lambda i 0.328 3 6 4.099 0.067 
Pillai Trace | 0.672 3 6 4.099 0.067 
Hotelling-Lawley Trace ! 2.050 3 6 4.099 0.067 


In the within-subjects table, you see that the trial effect is highly significant 

(F-ratio = 152.1, p-value < 0.0005). Below that table, we see that the linear trend 
across trials (Polynomial Order 1) is highly significant (F-ratio = 247.8, 

p-value < 0.0005). The hypothesis sum of squares for the linear, quadratic, and cubic 
polynomials sum to the total hypothesis sum of squares for trials (that is, 984.15 + 6.75 
+ 0.60 = 991.5). Notice that the total sum of squares is 991.5, while that for the linear 
trend is 984.15. This means that the linear trend accounts for more than 99% of the 
variability across the four trials. The assumption of compound symmetry is not 
required for the test of linear trend—so you can report that there is a highly significant 
linear decrease across the four trials (F-ratio = 247.8, p-value < 0.0005). 


Example 12 
Repeated Measures ANOVA for Two Trial Factors 


Repeated Measures enables you to handle several trial factors, so we include an 
example with two trial factors. It is an experiment from Winer, Brown and Michels 
(1991), which has one grouping factor (NOISE) and two trials factors (PERIODS and 
DIALS). The trial factors must be sorted into a set of dependent variables (one for each 
pairing of the two factors groups). It is useful to label the levels with a convenient 
mnemonic. The file is set up with variables P/D/ through P3D3. Variable P1D2 
indicates a score in the PERIODS = 1, DIALS = 2 cell. The data are in the file 
REPEAT2. 
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The input is: 


ANOVA 
USE REPEAT2 
CATEGORY NOISE 
DEPEND P1D1 .. P3D3 / REPEAT-3,3 NAMES-'period', 'dial' 
PLENGTH MEDIUM 
ESTIMATE 


Notice that REPEAT specifies that the two trial factors have three levels each. ANOVA 
assumes the subscript of the first factor will vary the slowest in the ordering of the 
dependent variables. If you have two repeated factors (DAY with four levels and 
AMPM with two levels), you should select eight dependent variables and type 
Repeat-4, 2. The repeated measures are selected in the following order: 


DAYl AM DAYl PM DAY2 AM DAY2 PM DAY3 AM DAY3 PM DAY4 AM 
DAY4 PM 


From this indexing, it generates the proper main effects and interactions. When more 
than one trial factor is present, ANOVA lists each dependent variable and the 
associated level on each factor. 


The output is: 
Dependent Variable Means 
P1Dl P1D2 P1D3 P2D1 P2D2 
748.000 52.000 63.000 37.167 42.167 
Dependent Variable Means 
P2D3 P3D1 P3D2 P3D3 
754.167 27,000 32.500 42.500 
Univariate and Multivariate Repeated Measures Analysis 


Between Subjects 


Source | ss df F-ratio p-value 
-------- * 

NOISE } 468.167 1 468.1 0.752 0.435 
Error | 2491.111 4 622.718 


Within Subjects 


Source df Mean Squares F-ratio p-value G-G H-F 


period » 2 1861.167 63.389 0.000 0.000 
period*NOISE | 333.000 2 166.500 5.671 0.029 0.057 0.029 
Error | 234,889 8 29.361 


Greenhouse-Geisser Epsilon | 
Huynh-Feldt Epsilon 1 1.000 
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Source i SS df Mean Squares  F-ratio p-value G-G H-F 

e A A A Se 
dial | 2370.333 2 1185.167 89.823 0.000 0.000 0.000 
dial*NOISE | 50.333 2 25.167 1.907 0.210 0.215 0.210 
Error i 105.556 8 13.194 


Greenhouse-Geisser Epsilon ; 0.917 
Huynh-Feldt Epsilon i 1.000 


Within Subjects 
Ss df 


E F-ratio 


pon dial 10.667 4 0.336 0.850 0.729 0.850 
period*dial*NOISE | 11.333 4 2.833 0.357 0.836 0.716 0.836 
Error į 127.111 16 7.944 

Greenhouse-Geisser Epsilon | 0.513 

Huynh-Feldt Epsilon 1 1.000 


Single Degree of Freedom Polynomial Contrasts 
Polynomial Test of Order 1 (Linear) 


Source ss df Mean Squares F-ratio p-value 
period + 3721.000 1 3721.000 73.441 0.001 
period*NOISE i 225.000 1 225.000 4.441 0.103 
Error + 202.667 4 50.667 

dial | 2256.250 1 2256.250 241.741 0.000 
dial*NOISE n 6.250 1 6.250 0.670 0.459 
Error | 37.333 4 9.333 

period*dial i 0.375 1 0.375 0.045 0.842 
period*dial*NOISE | 1.042 1 1.042 0.125 0.742 
Error i 33.333 4 8.333 

Polynomial Test of Order 2 (Quadratic) 

Source SS df Mean Squares F-ratio p-value 
period i 1.333 1 1.333 0.166 0.705 
period*NOISE ; 108.000 1 108.000 13.407 0.022 
Error |! 32.222 4 8.056 

dial t 114.083 1 114.083 6.689 0.061 
dial*NOISE i 44.083 1 44.083 2.585 0.183 
Error i 68.222 4 17.056 

period*dial i 3.125 H 3.125 0.815 0.418 
period*dial*NOISE | 0.125 1 0.125 0.033 0.865 
Error | 15.333 4 3.833 


Polynomial Test of Order 3 (Cubic) 


Source 


—-————À————  — + 
H 


Mean Squares 


period*dial 
period*dial*NOISE 
Error 


Polynomial Test of Order 4 


SS df Mean Squares  F-ratio p-value 


period*dial 1.042 $ 1.042 0.091 0.778 
period*dial*NOISE 7.042 1 7.042 0.615 0.477 
Error 45.778 4 11.444 
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Multivariate Repeated Measures Analysis 


Test of: period 


Statistic Value Hypothesis df Error df  F-ratio p-value 


Wilks's Lambda 2 3 28.145 0.011 
Pillai Trace | 0.949 2 3 28.145 0.011 
Hotelling-Lawley Trace | 18.764 2 3 28.145 0.011 


Test of: period*NOISE 


Hypothesis df Error df 


Statistic 

Wilks's Lambda 

Pillai Trace 
Hotelling-Lawley Trace 


CTI DES 


Test of: dial 


Statistic Error df FERES p-value 


Wilks's Lambda 
Pillai Trace 
Hotelling-Lawley Trace 


M (pes 


Test of: dial*NOISE 
Error df F-ratio 


s 
D 
A 
e 
o 


Statistic 


Wilks's Lambda 


Pillai Trace 0.435 2 3 AU pases 
Hotelling-Lawley Trace | 0.770 2 3 1.155 
Test of: period*dial 

Statistic H Value Hypothesis df 

a ease A arre tatem aiti - 
Wilks's Lambda 0.001 4 1 

Pillai Trace ub 999 A i 


331.445 0.041 


Hotelling-Lawley Trace 


Test of: period*dial*NOISE 


Error df F-ratio p-value 


Statistic 


Wilks's Lambda ] $ f 
Pillai Trace 1 . $ 
Hotelling-Lawley Trace | 2327.500 4 1 581.875 0.031 


The input is: 


GLM 
USE REPEAT2 
CATEGORY NOISE 
.. P3D3 = CONSTANT + NOISE / REPEAT=3,3, 
n NN a, NAMES- period' , 'dial' 
PLENGTH MEDIUM 
ESTIMATE 
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Example 13 
Repeated Measures Analysis of Covariance 


To do repeated measures analysis of covariance, where the covariate varies within 
subjects, you would have to set up your model like a split plot with a different record 
for each measurement. 

This example is from Winer, Brown and Michels (1991). This design has two trials 
(DAY! and DAY2), one covariate (AGE), and one grouping factor (SEX). The data are 
in the file WINER. 


The input is: 


ANOVA 
USE WINER 
CATEGORY SEX 
DEPEND DAY(1 .. 2) / REPEAT NAME='day' 
COVARIATE AGE 
ESTIMATE 


The output is: 
Dependent Variable Means 
DAY (1) DAY (2) 
716.500 — 11.875 
Univariate Repeated Measures Analysis 
Between Subjects 


Source | ss df. Mean Squares F-ratio p-value 

SEX 44.492 3.629 0.115 

AGE i 1 166.577 13.587 0.014 

Error | 61.298 5 12.260 

Within Subjects 

Source | SS df Mean Squares  F-ratio  p-value  G-G  H-F 

1 22.366 17.899 dT oe 

i 1 0.494 0.395 0.557 

day*AGE | 1 0.127 0.102 0.763 

Error 5 1.250 


Greenhouse-Geisser Epsilon | . 
Huynh-Feldt Epsilon [ 


The F-ratio for the covariate and its interactions, namely AGE (13.587) and 


DAY * AGE (0.102), are not ordinarily published; however, they help you understand 
the adjustment made by the covariate. 
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This analysis did not test the homogeneity of slopes assumption. If you want to test 
the homogeneity of slopes assumption, run the following model in GLM first: 


MODEL day(1 .. 2) = CONSTANT + sex + age + sex*age / REPEAT 
Then check to see if the SEX * AGE interaction is significant. 


To use GLM: 


GLM 
USE WINER 
CATEGORY SEX 
MODEL DAY(1 .. 2) = CONSTANT + SEX + AGE / REPEAT NAME='day' 
ESTIMATE 


Computation 


Algorithms 


Centered sum of squares and cross-products are accumulated using provisional 
algorithms. Linear systems, including those involved in hypothesis testing, are solved 
by using forward and reverse sweeping (Dempster, 1969). Eigensystems are solved 
with Householder tridiagonalization and implicit QL iterations. For further 
information, see Wilkinson and Reinsch (1971) or Chambers (1977). 
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Linear Models III:General Linear 
Models 


Leland Wilkinson and Mark Coward 


General Linear Model (GLM) can estimate and test any univariate or multivariate 
general linear model, including those for multiple regression, analysis of variance or 
covariance, and other procedures such as discriminant analysis and principal 
components. With the general linear model, you can explore randomized block 
designs, incomplete block designs, fractional factorial designs, Latin square designs, 
split plot designs, crossover designs, nesting, and more. The model is: 


Y =XB+e 


where Y is a vector or matrix of dependent variables, X is a vector or matrix of 
independent variables, B is a vector or matrix of regression coefficients, and e is a 
vector or matrix of random errors. See Searle (1971), Winer, Brown and Michels 
(1991), Kutner et al. (2004), or Cohen (2002) for details. 

Moreover, GLM also features the means model for missing cells designs. Widely 
favored for this purpose by statisticians (Hocking, 1985; Milliken and Johnson, 1984; 
Searle,1987), the means model allows tests of hypothesis in missing cells designs 
(using what are often called Type IV sum of squares). Furthermore, the means model 
allows direct tests of simple hypotheses (for example, within levels of other factors). 
Finally, the means model allows easier use of population weights to reflect 
differences in subclass sizes. 

The GLM module provides fifteen tests for pairwise comparisons based on the 
structure of data and the error rate to be controlled. The pairwise comparison tests are 
commonly named as post hoc tests; here tests are determined based on the 
assumptions on variance, viz., equal or unequal variances. One can use post hoc tests 
after fitting the model to check the differences between pairs of means. 
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In multivariate models, Y is a matrix of continuous measures. The X matrix can be 
either continuous or categorical dummy variables, according to the type of model. For 
discriminant analysis, X is a matrix of dummy variables, as in analysis of variance. For 
principal components analysis, X is constant (a single column of 1’s). For canonical 
correlation, X is usually a matrix of continuous right-hand variables (and Y is the 
matrix of left-hand variables). 

For some multivariate models, it may be easier to use ANOVA, which can handle 
models with multiple dependent variables and zero, one, or more categorical 
independent variables (that is, only the constant is present in the former). ANOVA 
automatically generates interaction terms for the design factor. 

SYSTAT offers three tests for checking normality: Kolmogorov-Smirnov 
(Lilliefors), Anderson-Darling, and Shapiro-Wilk test; and Levene's test for checkin g 
the homogeneity of variances. You can select any ofthe three types of sum of squares: 
Type I, Type II and Type III, for the analysis. 

After the parameters of a model have been estimated, they can be tested by any 
general linear hypothesis of the following form: 


ABC'-D 


where A is a matrix of linear weights on coefficients across the independent variables 
(the rows of B), C is a matrix of linear weights on the coefficients across dependent 
variables (the columns of B), B is the matrix of regression coefficients or effects, and 
D is a null hypothesis matrix (usually a null matrix). 

For the multivariate models described in this chapter, by default the C matrix is an 
identity matrix, and the D matrix is null. The A matrix can have several different forms 
but these are all submatrices of an identity matrix and are easily formed. 

The A matrix, C matrix, and D matrix are available for hypothesis testing in 
multivariate models. You can test parameters of the multivariate model estimated or 
factor the quadratic form of your model into orthogonal components. 

Resampling procedures are available in this feature. 
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General Linear Models in SYSTAT 


Model Estimation (in GLM) 


To specify a general linear model using GLM, from the menus choose: 


Analyze 
General Linear Model (GLM) 
Estimate Model... 


Analyze: General Linear Model Estimate Model 


Available variable(s]: 
WEIGHT 
1D 
SALBEG 
SEX 
TIME 
AGE 
SALNOW 
EDLEVEL 
WORK 
JOBCAT 
MINORITY 
SEXRACE 


Model options 
[V] Include constant 


Mear 


You can specify any multivariate linear model with General Linear Model. You must 
select the variables to include in the desired model. 
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Dependent(s). The variable(s) you want to examine. The dependent variable(s) should 
be continuous numeric variables (for example, INCOME). 


Independent(s). Select one or more continuous or categorical variables ( grouping 
variables). Independent variables that are not denoted as categorical are considered 
covariates. Unlike ANOVA, GLM does not automatically include and test all 
interactions. With GLM, you have to build your model. Suppose you want interactions 
or nested variables in your model, you need to build these components using the Cross 
and Nest buttons. To include lower-order effects with the interaction term, use the # 
button; e.g., A # B =A +B + A*B. 


Model options. The following model options allow you to include a constant in your 
model, do a means model, specify the sample size, and weight cell means. 


m Include constant. The constant is an optional parameter. Deselect Include constant 
to obtain a model through the origin. When in doubt, include the constant. 


m Means. Specifies a fully factorial design using means coding. This option is 
available when the model does not contain a constant, and contains at least one 
categorical variable. 


m Cases. When your data file is a symmetric matrix, specify the sample size that 
generated the matrix. 


m Weight. Weights cell means by the cell counts before averaging for the Means 
model. 


Save. Saves residuals and other data to a new data file. The following alternatives are 
available: 


= Residuals. Saves predicted values, residuals, Studentized residuals, and the 
standard error of predicted values, 


m Residuals/Data. Saves the statistics given by Residuals, plus all the variables in the 
working data file, including any transformed data values. 


m Adjusted. Saves adjusted cell means from analysis of covariance. 


Adjusted/Data. Saves adjusted cell means plus all the variables in the working data 
file, including any transformed data values. 


Coefficients. Saves the estimates of the Tegression coefficients. 
Model. Saves statistics given in Residuals and the variables used in the model. 
Partial. Saves partial residuals for univariate model, 
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m Partial/Data. Saves partial residuals plus all the variables in the working data file, 
including any transformed data values. 


Category 


You can specify numeric or character-valued categorical (grouping) variables that 
define cells. You want to categorize an independent variable when it has several 
categories such as education levels, which could be divided into the following 
categories: less than high school, some high school, finished high school, some 
college, finished bachelor's degree, finished master's degree, and finished doctorate. 
On the other hand, a variable such as age in years would not be categorical unless age 
were broken up into categories such as under 21, 21-65, and over 65. 


To specify categorical variables, click the Category tab in the GLM: Estimate Model 
dialog box. 
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Analyze: General Linear Model: Estimate Model 


| Model] Category | Repeated Measures! Estimation] Options | Resampling! 


Available variable(s}: 
| SEX 


Missing values. Specifies the cases with missing values for the categorical variable(s) 
to be included as a separate category in the analysis, 


Coding. You can select to use one of two different coding methods: 


= Dummy. Produces dummy codes for the design variables instead of effect codes. 
Coding of dummy variables is the classic analysis of variance parameterization, in 
which the sum of effects estimated for a classifying variable is 0. If your categorical 
variable has k categories, k — 1 dummy variables are created. 


m Effect. Produces parameter estimates that are differences from group means. 
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Repeated Measures 


In a repeated measures design, the same variable is measured several times for each 
subject (case). A paired-comparison t-test is the most simple form of a repeated 
measures design (for example, each subject has a before and after measure). 


SYSTAT derives values from your repeated measures and uses them in general linear 
model computations to test changes across the repeated measures (within subjects) as 
well as differences between groups of subjects (between subjects). Tests of the within- 
subjects values are called Polynomial Tests of Order 1, 2,..., up to k, where k is one 
less than the number of repeated measures. The first polynomial is used to test linear 
changes: Do the repeated responses increase (or decrease) around a line with a 
significant slope? The second polynomial tests if the responses fall along a quadratic 
curve, etc. 


To perform repeated measures analysis, click the Repeated Measures tab in the GLM: 
Estimate Model dialog box. 


nam 
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Analyze: General Linear Model: Estimate Model 


Suppose you select Perform repeated measures analysis, SYSTAT treats the 
dependent variables as a set of repeated measures. Optionally, you can assign a name 
for each set of repeated measures, specify the number of levels, and specify the metric 
for unevenly spaced repeated measures. 


Name. Name that identifies each set of repeated measures. 


Levels. Number of repeated measures in the set. For example, suppose you have three 
dependent variables that represent measurements at different times, the number of 
levels is 3. 


Metric. Metric that indicates the spacing between unevenly spaced measurements. For 
example, if measurements were taken at the third, fifth, and ninth weeks, the metric 
would be 3, 5, 9. 
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Estimation 


The Estimation tab allows you to specify a tolerance and confidence level, You can 
select complete or stepwise estimation procedures and specify entry and removal 
criteria. 


To specify estimation options, click the Estimation tab in the GLM: Estimate Model 
dialog box. 


Analyze: General Linear Model: Estimate M 
[Model] Categon| Repeated Meosus] Estimation [cji | recess 


| es 
Tolerance: [1e-012 | Stepwise options: 


enun 
Estimation 


The following options can be specified: 


Tolerance. Prevents the entry of a variable that is highly correlated with the 
independent variables already included in the model. Enter a value between 0 and 1. 
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Typical values are 0.01 or 0.001. The higher the value (closer to 1), the lower the 
correlation required to exclude a variable. 


Confidence. Specify a confidence level for the confidence interval for the regression 
coefficients. The default level is 0.95. 

Estimation. Controls the method used to enter and remove variables from the equation. 
= Complete. All independent variables are entered in a single step. 

W Stepwise. Variables are entered into or removed from the model, one at a time. 


Mixture model. Constrains the independent variables to sum to a constant, when the 
Complete estimation option is chosen. 


Stepwise options. The following alternatives are available for stepwise entry and 

removal: 

= Backward. Begins with all candidate variables in the model. At each step, 
SYSTAT removes the variable with the largest Remove value. 

= Forward. Begins with no variables in the model. At each step, SYSTAT adds the 
variable with the smallest Enter value. 


= Automatic. For Backward, at each step, SYSTAT automatically removes a variable 
from your model. For Forward, at each step, SYSTAT automatically adds a 
variable to the model. 


" Interactive. At each step in the model building, you select the variable to enter into 
or remove from the model. 


You can also control the criteria used to enter and remove variables from the model: 

m Probability. Specify probabilities to enter and to remove variable(s) from the 
model. A variable is entered into the model if its alpha value is less than the 
specified Enter value and is removed from the model if its alpha value is greater 
than the specified Remove value. Specify values between 0 and 1. 

m F-statistic. Specify F-to-enter and F-to-remove limits. Variables with F-statistic 
greater than the specified value are entered into the model if Tolerance permits and 
variables with F-statistic less than the specified value are removed from the model. 

m MaxStep. Specify the maximum number of steps. 


m Force. Forces the first n variables listed in your model to remain in the equation. 
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Options 


To specify the options, click the Options tab in the General Linear Model: Estimate 
Model dialog box. 


Analyze: General Linear Model Estimate Model 


Model!) Category | Repeated Measures Estimation] Options | Resampling) 


Assumptions check 
Normality tests 


Equality of variances tests 


O Levene 


Sums of squares 
O Type I: Sequential 

| O Type ll: Partially sequential 

| © Type lil: Adjusted 


Assumptions check. This provides options to check the basic assumptions of GLM. 


Normality tests. You can use the following normality tests to check the basic statistical 
assumption of GLM, normality of residuals. 


m Kolmogorov-Smirnov (Lillefors). It is a nonparametric test used for large 
samples. It is applied to continuous distributions and gives greater importance to 
the observations in the centre than those at the tails. 
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= Shapiro-Wilk. The test provides the Shapiro-Wilk test statistic and p-value for the 
selected dependent variable. The smaller the p-value, the worse is the fit. 


m Anderson-Darling. Anderson-Darling test is a standard goodness of fit test. It 
gives greater importance to the observations in the tails than those at the center. 

Equality of variances test. You can use the following equality of variances test to 

check the homogeneity of variances across all levels of the factors: 

= Levene's. The Levene's test is less sensitive than the Bartlett test to departures from 
normality. This test is an alternative to the Bartlett test. 

Sum of squares. For the model, you can choose a particular type of sum of squares. 

Type III is the one most commonly used and is the default. 

m Type I: Sequential. Uses type I sum of squares for the analysis. 

= Type II: Partially sequential. Uses type II sum of squares for the analysis. 


m Type III: Adjusted. Uses type III sum of squares for the analysis. This is the 
default. 


Hypothesis Test 


Contrasts are used to test relationships among cell means. The post hoc tests in GLM: 
Pairwise Comparisons are of the most simple form because they compare two means 
at a time. However, general contrasts can involve any number of means in the analysis. 


To test hypotheses, from the menus choose: 


Analyze 
General Linear Model (GLM) 
Hypothesis Test... 


11-187 


Linear Models III: General Linear Models 


Analyze:General Linear Model (GLM):Hypothesis Test 


| Main | Options | peci | Contrast | A Mat 2! CMatix| D Matrix) 


Hypothesis: | Effects M 


| Available effect(s): £i Selectedeffect(s) — — —— 
SEX I| Add» JOBCAT | 
[sa 
| Constant Gross =>] 


| . Emor term 


© Model MSE 


Contrasts can be defined across the categories of a grouping factor or across the levels 


of a repeated measure. 


Hypothesis. Select the type of hypothesis. The following choices are available: 


Model. Tests for the coefficients of the model. This is the default. 
All. Tests all the effects in the model. 

Effects. Tests jointly for the effects in the Selected effect(s) list. 
Specify. Tests the hypotheses in the Specify tab. 

A Matrix. Tests the hypotheses corresponding to the A Matrix tab. 


Within. Select the repeated measures factor across whose levels the contrast is defined. 


Error term. You can specify which error term to use for the hypothesis tests. 


m Model MSE. Uses the mean square error from the general linear model that you 


ran. 
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m MSE and df. Uses the mean square error and degrees of freedom you specify. Use 
this option if you know them from a previous model. 


m Between-subjects effect(s). Uses the main effect or interaction effect that you select 
from the Between-subject(s) effect list. 


Options 


To specify hypothesis options, click the Options tab in the GLM: Hypothesis Test dialog 
box. 


Analyze:General Linear Model (GLM):Hypothesis Test 


Matrix type 
@sscp 

© Correlation 
O Covariance 


Rotate fist | ] | canonical factors 


E Save scores andresuts: | —  —  — a 


Priors. Prior probabilities for discriminant analysis. Type a value for each group, 
separated by spaces. These probabilities should add to 1. For example, suppose you 
have three groups, priors might be 0.5, 0.3, and 0.2. The prior option is available when 
you select a single grouping variable as the effect to be tested. 
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Standardize. You can standardize canonical coefficients using the total sample or a 
within-groups covariance matrix. 


m Within groups is usually used in discriminant analysis to make comparisons easier 
when measures are on different scales. 


m Sample is used in canonical correlation. 


Factor. In a factor analysis with grouping variables, factor the Hypothesis (between- 
groups) matrix or the Error (within-groups) matrix. This allows you to compute 
principal components on the hypothesis or error matrix separately, offering a direct 
way to compute principal components on residuals of any linear model you wish to fit. 
You can specify the matrix type as SSCP, Correlations, or Covariance. 


Rotate. Specify the number of components to rotate. 


Save scores and results. You can save the results to a SYSTAT data file. Exactly what 
is saved depends on the analysis. When you save scores and results, the extended 
output is automatically produced. This enables you to see more detailed output when 
computing these statistics. 


Specify 


To specify contrasts for between-subjects effects, select the Specify option of 
Hypothesis in the GLM: Hypothesis Test dialog box. The Specify tab gets enabled. 
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Analyze:General Linear Model (GLM):Hypothesis Test 


on T 
TIBI] - 0 
2*A[1- AD] 


You can use GLM's cell means "language" to define contrasts across the levels of a 
grouping variable in a multivariate model. For example, for a two-way factorial 
ANOVA design with D/SEASE (four categories) and DRUG (three categories), you 
could contrast the marginal mean for the first level of drug against the third level by 
specifying: 

DRUG[1] = DRUG[3] 


Note that square brackets enclose the value of the category. For string variables, their 
values are assumed to be in the upper case unless they are enclosed in quotes. For 
example, GENDER$[male] is read as GENDER$[MALE], whereas GENDERS[male'] will 
prompt SYSTAT to look for the exact string 'male'. For the simple contrast of the first 
and third levels of DRUG for the second disease only, specify: 


DRUG[1] DISEASE[2] = DRUG[3] DISEASE[2] 
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The syntax also allows statements like: 

-3*DRUG[1] - 1*DRUG[2] + 1*DRUG[3] + 3*DRUG [4] 
where the right-hand side is considered zero unless you specify a value for it or specify 
it through a D matrix. Fora univariate model, you can also choose one of the following: 
Pooled. Uses the error term from the current model. 


Separate. Generates a separate variances error term. 
Contrast 
Contrast generates a contrast for a grouping factor or a repeated measures factor. To 


specify contrasts, select an effect under the Effect option of Hypothesis or a repeated 
measures factor in the Within drop-down list. The Contrast tab gets enabled. 


Analyze:General Linear Model (GLM):Hypothesis Test 


| Mein | Options] 


Use contrast 
© Custom: 


O Adjacent Difference O Deviation 
O Helmet 

O Reverse Helmert 
O Polynomial - 


O Simple 


0.000 
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SYSTAT offers eight types of contrasts: 


Custom. Enter your own custom coefficients. For example, suppose your factor has 
four ordered categories (or levels), you can specify your own coefficients, such as —3 
—1 1 3, by typing these values in the Custom text box. 


Adjacent difference. Compares each level of the factor with its adjacent level. 


Helmert. Compares the mean of each level of the selected factor with the mean of the 
succeeding levels. 


Reverse Helmert. Compares the mean of each level of the selected factor with the 
mean of the previous levels. 


Polynomial. Generates orthogonal polynomial contrasts (to test linear, quadratic, or 
cubic trends across ordered categories or levels). 


= Order. Enter | for linear, 2 for quadratic, etc. 


= Metric. Use Metric when the ordered categories are not evenly spaced. For 
example, when repeated measures are collected at weeks 2, 4, and 8, enter 2,4,8 as 
the metric. 


Deviation. The deviation contrast compares the mean of the dependent variable for 
each level of the selected categorical variable (except a reference level) to the overall 
mean (grand mean) of the dependent variable. 


Simple. The simple contrast compares each level of the selected factor against the 
specified reference level. This type of contrast is useful when there is a control group. 
You can choose any level or category as the reference. 


Sum. In a repeated measures ANOVA, totals the values for each subject. 


A Matrix 


To specify an A matrix, select the A Matrix in the Hypothesis drop-down list of the 
GLM: Hypothesis Test dialog box. The A Matrix tab gets enabled. 
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Analyze:General Linear Model (GLM):Hypothesis Test 


conso] A Mati | C Matrix] D Mati 


“Main | Options 55 


ting the coefficient estimates (the rows of B). 
f the A matrix. The A matrix has as many 


cients (including the constant) in your model. 
ur hypothesis 


A is a matrix of linear weights contras! 
You can write your hypothesis in terms O 


columns as there are regression coeffi 
The number of rows in A determines how many degrees of freedom yo 


involves. 


€ Matrix 


d measures analysis of variance 


The C matrix is used to test hypotheses for repeate 
C has as many columns as there 


designs and models with multiple dependent variables. 
are dependent variables. By default, the C matrix is the identity matrix. 


To specify a different C matrix, click the C Matrix tab in the GLM: Hypothesis Test 


dialog box. 
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Í Main | Options | Sp=- 


[V] Use matrix: 


D Matrix 


D is a null hypothesis matrix (by default null matrix). The D matrix, if you use it, must 
have the same number of rows as A. For univariate multiple regression, D has only one 
column. For multivariate models (multiple dependent variables), the D matrix has one 
column for each dependent variable. 


To specify a different D matrix, click the D matrix tab in the GLM: Hypothesis Test 
dialog box. 
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Main |, Options), 


V| Use matrix: 


Toggling between the command line and GUI is supported in ANOVA, GLM, 
MANOVA, REGRESS, MIXED, LOGIT, LOGLINER, and RSM. That is, if 
estimation is performed through a dialog box, then post estimation analysis can be 


performed through commands and vice-versa. 


Pairwise Comparisons 


nd the treatment pairs which are significantly 
ments with their respective 
(mct) offered by SYSTAT under 


After fitting the model, one can fi 
different, or form several homogeneous sets of treat 
p-values by using several multiple comparison tests 
equal or unequal variance assumptions. 
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To open the Pairwise Comparisons dialog box, from the menus choose: 
Analyze 


General Linear Model (GLM) 
Pairwise Comparisons... 


Analyze:General Linear Model (GLM):Pairwise Comparisons ? [x] 


FTE m NM 
~ (hom | 

_ Cross 2 

AUCI 


E Duncan 
Dñe6wa 
(Fisher's LSD [DhHochberg's GT2 
DO Sidak C Gabriel 
E Scheffe [Student Newman-Keuls 
| C Tukey's b 
© Unequal variances 


Tamhane's T2 | | Games-Howell Dunnett's T2 


Confidence: [095] 


EE 


Groups. Select the variable that defines the groups. 


Tests. There are several post hoc tests to compare the means of the dependent variable 
for the selected grouping variable. 


Equal variances. Tests in this group assume equality of variances across all levels of 
the grouping variable. 


= Tukey. Uses the Studentized range distribution to make all pairwise comparisons. 
This is the default. 


® Bonferroni. Uses Student's z statistic. It sets the family-wise error rate as 
(1-Confidence)/(Total number of comparisons). 
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m Fisher's LSD. Equivalent to multiple f tests between all pairs of groups. The 
disadvantage of this test is that no attempt is made to adjust the observed 
significance level for multiple comparisons. 

Sidak. Uses Student's statistic for pairwise multiple comparisons. 

Scheffé. The significance level of Scheffé’s test is designed to allow all possible 
linear combinations of group means to be tested, not just pairwise comparisons 
available in this feature. The result is that Scheffé's test is more conservative than 
other tests. 

m Tukey’sb. Uses the Studentized range distribution. The critical value is the average 
of the corresponding values for the Tukey’s HSD test and the S-N-K test. 

= Duncan. Uses Studentized range distribution. It yields homogeneous subsets of 
group levels. 

m R-E-G-WQ. Ryan-Einot-Gabriel-Welsch Qtestisa modification of the S-N-K test 
where the critical values decrease as the range in the set being considered 
decreases. 

Hochberg’s GT2. Uses the Studentized maximum modulus distribution. 
Gabriel. Uses Studentized maximum modulus distribution. It is equivalent to the 
GT2 test for balanced ANOVA. 

m Student-Newman-Keuls. Uses Studentized range distribution. It yields 
homogenous subsets of group levels. 

m Dunnett. The Dunnett test is available only with one-way designs. Dunnett 
compares a set of treatments against a single control mean that you specify. You 
can choose from the following three alternative hypotheses: (a) 2-sided (not equal 
to), (b) less than, or (c) greater than control level. 2-sided is the default. 


Unequal variances. The following tests do not require the homogeneity of variance 

assumption.These tests use the Welsch procedure for determining the denominator 

degrees of freedom. 

m Tamhane’s T2. Uses the Student's ¢ distribution. Uses the Sidak inequality to find 
the alpha level. 

m Games-Howell. Uses the Studentized range distribution. 

m Dunnett's T3. Uses the Studentized maximum modulus distribution. 


Confidence. Specify confidence level for pairwise comparisons tests. The default value 
is 0.95. 
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Error term 


To specify the error term, click the Error Term tab in the GLM: Pairwise Comparisons 
dialog box. 


Analyze:General Linear Model (GLM):Pairwise Comparisons [? fX) 


| Main Error Term 


| 
© Model MSE 


© Between-subjects effect(s) 
Available effect(s): 
| SEX 
Constant 


Error term. You can choose one of the following: 


= Model MSE. Uses the mean square error (MSE) from the general linear model that 
you ran. 


= MSE and df. Uses the mean square error term and degrees of freedom that you 
specify. Use this option if you know them from a previous model. 


m Between-subjects effect(s). Select this option to use the main effect error term or 
the interaction error term in all the tests. 


Toggling between the command line and GUI is supported in ANOVA, GLM, 
MANOVA, REGRESS, MIXED, LOGIT, LOGLINER, and RSM. That is, if 
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estimation is performed through a dialog box then post estimation analysis can be 
performed through commands and vice-versa. 


Post hoc Tests for Repeated Measures 


After performing analysis of variance, we just have an F-ratio, which tells us that 
means are not equal--we still do not know exactly which means are significantly 
different from which other ones. Post hoc tests can only be used when the "omnibus" 
ANOVA found a significant effect. If the F-value for a factor turns out non-significant, 
you cannot go further with the analysis. This protects the post hoc test from being used 
too liberally. 


The main problem that designers of post hoc tests try to deal with is alpha 
inflation.This refers to the fact that the more tests you conduct at alpha=0.05, the more 
likely you are to claim you have significant result when you shouldn't have. The overall 
chance of a type I error rate in a particular experiment is referred to as the 
“experiment-wise error rate” (or family-wise error rate). 


To perform the Post hoc Test for Repeated Measures, from the menus choose: 


Analyze 
General Linear Model (GLM) 
Post hoc Test for Repeated Measures... 


Post hoc Test for Repeated Measures (? J| 


— 


Correction for multiple comparisons 
(9 None 

© Bonferroni 

O Sidak 


oK 


name. Select a factor name from the drop-down list of factors defined for the 


Factor 
model. 
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Correction for multiple comparisons. The following options are available: 


m Bonferroni. To keep the experiment-wise error rate to a specified level 
(alpha-0.05) a simple way is to divide the acceptable alpha level by the number of 
comparisons we intend to make. That is, for any one comparison to be considered 
significant, the obtained p-value would have to be less than alpha/(num of 
comparisons). Select this option to perform a Bonferroni correction. 


m Sidak. The experiment-wise error explained above is kept in control by the use of 
the formula: sidak alpha = 1-(1-alpha)"*, where c is the number of paired 
comparisons. 


Toggling between the command line and GUI is supported in ANOVA, GLM, 
MANOVA, REGRESS, MIXED, LOGIT, LOGLINER, and RSM. That is, if 
estimation is performed through a dialog box then post estimation analysis can be 
performed through commands and vice-versa. 


Using Commands 


Select the data with USE FILENAME and continue with: 


GLM 
MODEL varlist1=CONSTANT + varlist2 + varl *var2 +, var3 (var4) / 
REPEAT=m,n,.. REPEAT=m(x1 ;X2,—), n(y1,y2,..) 
NAMES=‘namel’ , ‘name2’ sm , MEANS, WEIGHT Nen 


CATEGORY grpvarlist / MISS EFFECT or DUMMY 
PLENGTH SHORT or MEDIUM or LONG 


SAVE filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 
‘comment’ 
WORK filename / COEF MODEL RESID DATA PARTIAL ADJUSTED 
‘comment’ 
ESTIMATE / NTEST = KS, SW, AD HTEST = LEVENE 
SS = TYPE1 or TYPE2 or TYPE3 QUICK or NOQUICK 
MIX TOL=n SAMPLE = BOOT(m,n), SIMPLE(m,n), JACK 


For stepwise model building, use START in place of ESTIMATE: 


START / BACKWARD or FORWARD TOL=n ENTER=p REMOVE=p, 
FENTER=n FREMOVE=n FORCE=n MAXSTEP=n 
STEP no argument or var or index / AUTO ENTER=p, REMOVE=p 


FENTER=n FREMOVE=n 
STOP / QUICK or NOQUICK 
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To perform hypothesis tests: 


HYPOTHESIS 
EFFECT varlist, varl&var2,.. 
WITHIN ‘name’ 
CONTRAST [matrix] / DIFFERENCE or SUM or DEVIATION [c] or 
SIMPLE[c] or HELMERT or RHELMERT or 
POLYNOMIAL ORDER-n  METRIC-m,n,.. 
SPECIFY hypothesis lang / POOLED or SEPARATE 
AMATRIX [matrix] 
CMATRIX [matrix] 
DMATRIX [matrix] 
ALL 
POST grpvar/ LSD BONF-n TUKEY SCHEFFE SIDAK SNK BTUKEY DUNCAN 
GT2 GABR QREG GH T2 T3 POOLED SEPARATE 
DUNNETT = LT or GT or TWO CONTROL = ‘levelname’ 
PAIRWISE 'factorname'/ BONF or SIDAK 
ROTATE n 
TYPE CORR or COVAR or SSCP 
STAND TOTAL or WITHIN 
FACTOR HYPOTHESIS or ERROR 
ERROR value(df) or var or varl*var2 or varl & var2 or matrix 
PRIORS m n p .. 
TEST/CONFI - n 


Usage Considerations 


Types of data. Normally, you analyze raw cases-by-variables data with GLM. You can, 
however, use a symmetric matrix data file (for example, a covariance matrix saved in 
a file from Correlations) as input. Suppose you use a matrix as input, you must specify 
a value for Cases when estimating the model (under the Model tab in the GLM: 
Estimate Model dialog box) to specify the sample size of the data file that generated 
the matrix. The number you specify must be an integer greater than two. 

Be sure to include the dependent as well as independent variables in your matrix. 
SYSTAT picks out the dependent variable you name in your model. 

SYSTAT uses the sample size to calculate degrees of freedom in hypothesis tests. 
SYSTAT also determines the type of matrix (SSCP, Covariance, and so on) and adjusts 
appropriately. With a correlation matrix, the raw and standardized coefficients are the 
same; therefore, you cannot include a constant when using SSCP, Covariance, or 
Correlation matrices. Because these matrices are centered, the constant term has 
already been removed. 

The triangular matrix input facility is useful for *meta-analysis" of published data 
and missing value computations; however, you should heed the following warnings: 
First, suppose you input correlation matrices from textbooks or articles, you may not 
get the same regression coefficients as those printed in the source. Because of round- 
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off error, printed and raw data can lead to different results. Second, suppose you use 
pairwise deletion with Correlations, the degrees of freedom for hypotheses will not be 
appropriate. You may not even be able to estimate the regression coefficients because 
of singularities. 

In general, correlation matrices containing missing data produce coefficient 
estimates and hypothesis tests that are optimistic. You can correct for this by 
specifying a sample size smaller than the number of actual observations (preferably set 
it equal to the smallest number of cases used for any pair of variables), but this is a 
guess that you can refine only by doing Monte Carlo simulations. There is no simple 
solution. Beware, especially, of multivariate regressions (MANOVA and others) with 
missing data on the dependent variables. You can usually compute coefficients, but 
hypothesis testing produces results that are suspect. 


Print options. GLM produces extended output if you set the output length to LONG or 
if you select Save scores and results in the GLM Hypothesis Test dialog box. 

For model estimation, the extended output adds the following: total sum of product 
matrix, residual (or pooled within groups) sum of product matrix, residual (or pooled 
within groups) covariance matrix, and the residual (or pooled within groups) 
correlation matrix. 

For hypothesis testing, the extended output adds A, C, and D matrices, the matrix 
of contrasts, and the inverse of the cross products of contrasts, hypothesis and error 
sum of product matrices, tests of residual roots, canonical correlations, coefficients, 
and loadings. 


Quick Graphs. If no variables are categorical, GLM produces Quick Graphs of 
residuals versus predicted values. For categorical predictors, GLM produces graphs of 
the least-squares means for the levels of the categorical variable(s). 


Saving files. Several sets of output can be saved to a file. The actual contents of the 
saved file depend on the analysis. Files may include estimated regression coefficients, 
model variables, residuals, predicted values, diagnostic statistics, canonical variable 
scores, and posterior probabilities (among other statistics). 


BY groups. Each level of a BY variable yields a separate analysis. However, for 
Hypothesis Testing, BY groups does not work. You have to resort to 
Data--> Select Cases commands. 


Case frequencies. GLM uses the FREQUENCY variable, if present, to duplicate cases. 


Case weights. GLM uses the values of any WEIGHT variable to weight each case. 
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Examples 


Example 1 
One-Way ANOVA 


The following data, KENTON, are from Kutner et al. (2004). The data comprise unit 
sales of a cereal product under different types of package designs. Ten stores were 
selected as experimental units. Each store was randomly assigned to sell one of the 
package designs (each design was sold at two or three stores). 


PACKAGE SALES 
12 
18 
14 
12 
13 
19 
17 
21 
24 
30 


4» 4 oU oU OU MOT M — = 


Numbers are used to code the four types of package designs; alternatively, you could 
have used words. Kutner et al. (2004) report that cartoons are part of designs 1 and 3 
but not designs 2 and 4; designs 1 and 2 have three colors; and designs 3 and 4 have 
five colors. Thus, string codes for PACKAGES might have been ‘Cart 3’, ‘NoCart 3”, 
“Cart 5°, and ‘NoCart 5”. Notice that the data does not need to be ordered by 
PACKAGE as shown here. 


The input is: 


GLM 
USE KENTON 
CATEGORY PACKAGE 
MODEL SALES=CONSTANT + PACKAGE 
GRAPH NONE 
ESTIMATE 
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The output is: 


Effects coding used for categorical variables in model. 
Categorical values encountered during processing are 


Variables 1 
"rcr gii NUN Ppa nica 


PACKAGE (4 levels) | 1.000 2.0 


Dependent Variable | SALES 
N 1 


Multiple R | 0.921 
Squared Multiple R } 0.849 


Analysis of Variance 


Source ss df Mean Squares F-ratio p-value 


PACKAGE | .000 3 86.000 11.217 0.007 
Error i 46.000 6 7.667 


This is the standard analysis of variance table. The F-ratio (11.217) appears 
significant, so you could conclude that the package designs differ significantly in their 
effects on sales, provided the assumptions are valid. 


Pairwise Multiple Comparisons 


SYSTAT offers fifteen methods for comparing pairs of means: Bonferroni, Tukey- 
Kramer HSD, Sidak, Duncan, Scheffé, Fisher’s LSD, Tukey’s B, R-E-G-W Q, 
Hochberg’s GT2, Tamhanne's T2, Games-Howell, Dunnett T3, Gabriel, Student- 
Newman-Keuls, and Dunnett’s test. 

The Dunnett test is available only with one-way designs. Dunnett requires the value 
ofa control group against which comparisons are made. By default, two-sided tests are 
computed. One-sided Dunnett tests are also available. Incidentally, for Dunnett’s tests 
on experimental data, you should use the one-sided option unless you cannot predict 
from theory whether your experimental groups will have higher or lower means than 
the control. 

Comparisons for the pairwise methods are made across all pairs of least-squares 
group means for the design term that is specified. For a multiway design, marginal cell 
means are computed for the effects specified before the comparisons are made. 

To determine significant differences, simply look for pairs with probabilities below 
your critical value (for example, 0.05 or 0.01). All multiple comparison methods 
handle unbalanced designs correctly. 

After you estimate your ANOVA model, it is easy to do post hoc tests. To do a 
Tukey-Kramer HSD test, first estimate the model, then specify these commands. 
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The input is: 


HYPOTHESIS 
POST PACKAGE / TUKEY 
TEST 


The output is: 

Post Hoc Test of SALES 

Using least squares means. 

Using model MSE of 7.667 with 6 df. 


Tukey's Honestly-Significant-Difference Test 


PACKAGE(i)  PACKAGE(j) Difference p-value 95.0% Confidence Interval 
Lower Upper 


Results show that sales for the fourth package design (five colors and no cartoons) are 
significantly larger than those for packages | and 2. None of the other pairs differ 
significantly. 


Contrasts 


This example uses two contrasts: 
m We compare the first and third packages using coefficients of (1, 0, —1, 0). 


m We compare the average performance of the first three packages with the last, 
using coefficients of (1, 1, 1, -3). 


The input is: 


HYPOTHESIS 

EFFECT PACKAGE 
CONTRAST [1 O -1 0] 
TEST 


HYPOTHESIS 

EFFECT PACKAGE 
CONTRAST [1 1 1 -3] 
TEST 


For each hypothesis, we specify one contrast, so the test has one degree of freedom; 
therefore, the contrast matrix has one row of numbers. These numbers are the same 
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ones you see in ANOVA textbooks, although ANOVA offers one advantage—you do not 
have to standardize them so that their sum of squares is 1. 


The output is: 
Test for effect called: PACKAGE 


A Matrix 


Test of Hypothesis 


Source t ss df Mean Squares F-ratio p-value 
EAC THEN ey I a Sa eed rovc. AR cie AR Ea 
Hypothesis | 19.200 1 19.200 2.504 0.165 
Error | 46.000 6 7.667 


Test for effect called: PACKAGE 
A Matrix 
0.000 4.000 4.000 4.000 
Test of Hypothesis 


Source i SS df Mean Squares F-ratio 
Phor pr NN he ai nen ae misc icai NR 
Hypothesis | 204.000 1 204.000 26.609 
Error | 46.000 6 7.667 


For the first contrast, the F-ratio (2.504) is not significant, so you cannot conclude that 
the impact of the first and third package designs on sales is significantly different. 
Incidentally, the A matrix contains the contrast. The first column (0) corresponds to the 
constant in the model, and the remaining three columns (1 0 —1) correspond to the 
dummy variables for PACKAGE. 

The last package design is significantly different from the other three taken as a 
group. Notice that the A matrix looks much different this time. Because the effects sum 
to 0, the last effect is minus the sum of the other three; that is, letting a; denote the 
effect for level i of package, 


a; +a, + az +a4=0 
so 
a4 =a] + a5 + 03) 


and the contrast is 
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a * &5 + 03-304 
which is 

a. + az + 03 301 — 85 - 03) 
which simplifies to 

4a, +40 + 405 


Remember, SYSTAT does all this work automatically. 


Orthogonal Polynomials 


Constructing orthogonal polynomials for between-group factors is useful when the 
levels of a factor are ordered. To construct orthogonal polynomials for your between- 
groups factors, the input is: 


HYPOTHESIS 

EFFECT PACKAGE 

CONTRAST / POLYNOMIAL ORDER=2 
TEST 


The output is: 


Test for effect called: PACKAGE 


A Matrix 


Hypothesis } 60. 
Error | 46. 6 7.667 


Make sure that the levels of the factor—after they are sorted by the procedure 
numerically or alphabetically—are ordered meaningfully on a latent dimension. 
Suppose you need a specific order, use LABEL or ORDER; otherwise, the results will 
not make sense. In the example, the significant quadratic effect is the result of the 


fourth package having a much larger sales volume than the other three. 
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Effect and Dummy Coding 


The effects in a least-squares analysis of variance are associated with a set of dummy 
variables that SYSTAT generates automatically. Ordinarily, you do not have to concern 
yourself with these dummy variables; suppose you want to see them, you can save 
them in to a SYSTAT file. 


The input is: 


GLM 
USE KENTON 
CATEGORY PACKAGE 
MODEL SALES=CONSTANT + PACKAGE 
GRAPH NONE 
SAVE MYCODES / MODEL 
ESTIMATE 
USE MYCODES 
FORMAT 12,0 
LIST SALES x( 1.. 3) 


The listing of the dummy variables follows. 


The output is: 
Case | SALES X(1) X(2) X(3) 
peo Misco AME tit cia 
1 1 12 1 0 0 
2 18 1 0 0 
3 H 14 0 1 0 
4 i 12 0 1 0 
5 1 13 0 1 0 
EE 0 0 1 
F 17 0 0 1 
B 21 0 0 1 
9 i 24 -1 -1 -1 
10 t 30 -1 “1 AL 


The variables X(1), X(2), and X(3) are the effects-coding dummy variables generated 
by the procedure. All cases in the first cell are associated with dummy values 1 0 0; 
those in the second cell with 0 1 0; the third, 0 0 1; and the fourth, —1 —1 —1. Other least- 
squares programs use different methods to code dummy variables. The coding used by 
SYSTAT is the one most widely used and guarantees that the effects sum to 0. 
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If you had used dummy coding, these dummy variables would be saved: 


SALES X(1) X(2) X(3) 


12 1 0 0 
18 1 0 0 
14 0 1 0 
12 0 1 0 
13 0 1 0 
19 0 0 0 
19 0 0 1 
17 0 0 1 
21 0 0 1 

0 0 0 

0 0 0 


This coding yields parameter estimates that are the differences between the mean for 
each group and the mean of the last group. 


Example 2 
Analysis of Covariance (ANCOVA) 


Winer, Brown and Michels (1991) use the COVAR data file for an analysis of 
covariance in which X is the covariate and TREAT is the treatment. Cases do not need 
to be ordered by the grouping variable TREAT. 

To define an ANCOVA model in GLM, we have to select factors (TREAT) and 
covariates (X) as independent variables and define only factors as categorical 
variable(s). SYSTAT automatically assumes non-categorical variables as covariates. 


The input is: 


GLM 
USE COVAR 
CATEGORY TREAT 
MODEL Y = CONSTANT + TREAT + X + TREAT*X 
ESTIMATE 
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The output is: 
Dependent Variable } Y 
N i 21 
Multiple R ! 0.921 


Squared Multiple R 
Analysis of Variance 


Source Type III SS df Mean Squares F-ratio 


i 2 

i 1 15.672 * 

i 0.667 2 0.334 0.519 
Error i 9.635 15 0.642 


The probability value for the treatment by covariate interaction is 0.605, so the 
assumption of homogeneity of slopes is justifiable. 


Testing Interaction Contrasts 


One might be interested in testing the interaction effect for different levels of treatment. 
SYSTAT does not support interaction contrasts directly through the CONTRAST 
command but you can test this by using the AMATRIX command. 


The input is: 


NOTE “INTERACTION CONTRAST [1 0 -1]" 
HYPOTHESIS 

AMATRIX [0 0 0 0 2 1] 

TEST 


NOTE "INTERACTION CONTRAST [-2 1 1]" 
HYPOTHESIS 


AMATRIX [0 0 O 0 -3 0] 
TEST 


The output is: 


Interaction Contrast [1 0 -1] 
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Test of Hypothesis 


Source E ss df Mean Squares F-ratio p-value 

Bicis NN LEUR PING io AS 
Hypothesis | 0.558 1 0.558 0.868 0.366 
Error i 9.635 15 0.642 


0.000 0.000 0.000 0.000 -3.000 


A Matrix 


Mean Squares F-ratio p-value 


Notice that the interaction contrast matrix and the A Matrix are different. Refer 
Example 1 in “Contrasts on page 205 


Example 3 
Randomized Block Designs 


^ randomized block design is like a factorial design without an interaction term. The 
following example is from Kutner et al. (2004). Five blocks of judges were given the 
task of analyzing three treatments. Judges are stratified within blocks, so the 
interaction of blocks and treatments cannot be analyzed. These data are in the file 


BLOCK. 


The input is: 


GLM 
USE BLOCK 
CATEGORY BLOCK, TREAT 
MODEL JUDGMENT = CONSTANT + BLOCK + TREAT 


ESTIMATE 


You must use GLM instead of ANOVA because you do not want the BLOCK*TREAT 


interaction in the model. 
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Dependent Variable | JUDGMENT 
N t 15 
Multiple R ! 0.970 
Squared Multiple R ! 0.940 
Analysis of Variance 
Source ; Type III SS df Mean Squares  F-ratio p-value 
E NE eod 
BLOCK 171.333 4 42.833 14,358 0.001 
TREAT | 202.800 2 101.400 33.989 0.000 
Error | 23.867 8 2.983 
Example 4 


Incomplete Block Designs 


Randomized blocks can be used in factorial designs. Here is an example from John 
(1971). The data (in the file JOHN) involve an experiment with three treatment factors 
(A, B, and C) plus a blocking variable with eight levels. Notice that data were collected 
on 32 of the possible 64 experimental situations. 


BLOCK A B € ¥ BLOCK A B C Y 
1 1 1 1 101 5 1 I 1 87 
1 2 1 2 373 3 2 1 2 324 
1 1 2 2 398 5 1 pA l| 279 
1 2 2 1 291 5 2 2 2 47 
2 1 1 2 312 6 1 1 2.1323 
2 2 1 1 106 6 2 1 l| 128 
2 1 2 1 265 6 1 2 2 423 
2 2 2 2 450 6 2 2 | 334 
3 1 1 1 106 7 1 1 I (M 
3 2 2 1 306 rj 2 1 1 103 
3 1 1 2 324 7 1 2 2 445 
3 2 2 2 449 7 2 2 2 437 
4 1 2 1 272 8 1 1 2 324 
4 2 1 1 89 8 2 1 2 361 
4 1 2 2 407 8 1 2 1 30 
4 2 1 2 338 8 2 2 1 21 
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The input is: 


GLM 
USE JOHN 
CATEGORY BLOCK, A, B, C 
MODEL Y = CONSTANT + BLOCK + AS B & C 


ESTIMATE 

The output is: 
Dependent Variable | X 
N pets — 
Multiple R | 0.994 


Squared Multiple R | 0.988 


Analysis of Variance 


Source | Type III SS df Mean Squares F-ratio p-value 
Eris NN stus E cU tH ict EE 
BLOCK | 2638.469 7 376.924 1.182 0.364 
A 1 3465.281 1 3465.281 10.862 0.004 
B | 161170.031 1 161170.031 505,209 0.000 
c į 278817.781 1 278817.781 873.992 0.000 
A*B i 28.167 1 28.167 0.088 0.770 
A*C i 1802.667 1 1802.667 5.651 0.029 
B*C i 11528.167 1 11528.167 36.137 0.000 
A*B*C | 45.375 1 45.375 0.142 0.711 
Error | 5423.281 17 319.017 
Example 5 


Fractional Factorial Designs 


Sometimes a factorial design involves so many combinations of treatments that certain 
cells must be left empty to save experimental resources. At other times, a complete 
randomized factorial study is designed, but loss of subjects leaves one or more cells 
completely missing. These models are similar to incomplete block designs because not 
all effects in the full model can be estimated. Usually, certain interactions must be left 
out of the model. 

The following example uses some experimental data that contain values in only 8 
out of 16 possible cells. Each cell contains two cases. The pattern of non-missing cells 
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makes it possible to estimate only the main effects plus three two-way interactions. The 


data are in the file FRACTION. y 
ok Clie D Y 
1 1 1 7 
1 1 1 1 3 
o | 1 1 1 
TM 1 1 2 
2 1.2 ae a | 12 
2 1.24 | 13 
1 23202 1 14 
1 2 2 1 15 
2 1 [44 8 
2 1 $2 6 
1 2-472 12 
1 23 (uis, 2 10 
1 Pi he Be) 6 
1 pe ee 4 
ee 6 
E MR Bark | 7 
The input is: 
GLM X 
USE FRACTION 


CATEGORY A, B, C, D 
MODEL Y = CONSTANT + A + B + C « D 4 A*B + A*C 4 BEC 
ESTIMATE 


We must use GLM instead of ANOVA to omit the higher-way interactions that ANOVA 
automatically generates. 


The output is: 
Dependent Variable 
N 


Multiple R 10 
Squared Multiple R 
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Analysis of Variance 


df Mean Squares  F-ratio p-value 


1 16.000 8.000 0.022 
1 4.000 2.000 0.195 
1 49.000 24.500 0.001 
H 4.000 2.000 0.195 
1 182.250 91.125 0.000 
1 12.250 6.125 0.038 
1 2.250 1.125 0.320 
8 2.000 


When missing cells turn up by chance rather than by design, you may not know which 
interactions to eliminate. When you attempt to fit the full model, SYSTAT informs you 
that the design is singular. In that case, you may need to try several models before 
finding an estimable one. It is usually best to begin by leaving out the highest-order 
interaction (A*B*C*D in this example). Continue with subset models until you get an 
ANOVA table. 

Looking for an estimable model is not the same as analyzing the data with stepwise 
regression because you are not looking at p-values. After you find an estimable model, 
stop and settle with the statistics printed in the ANOVA table. 


Example 6 
Nested Designs 


Nested designs resemble factorial designs with certain cells missing (incomplete 
factorials). This is because one factor is nested under another, so not all combinations 
of the two factors are observed. For example, in an educational study, classrooms are 
usually nested under schools because it is impossible to have the same classroom 
existing at two different schools (except as antimatter). The following example (in 
which teachers are nested within schools) is from Kutner et al. (2004). The data 


(learning scores) look like this: 


TEACHER1 TEACHER2 


SCHOOL! 25 14 
29 1 
SCHOOL2 11 22 
6 18 
SCHOOL3 17 5 


20 2 
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In the study, there are actually six teachers, not just two; thus, the design really looks 1 


like this: 
TEACHER! TEACHER2 TEACHER3 TEACHER4 TEACHERS TEACHER6 
SCHOOL1 25 14 
29 11 
SCHOOL2 11 22 
6 18 
SCHOOL3 17 5 


The data are set up in the file SCHOOLS 
TEACHER SCHOOL LEARNING 


1 1 25 
1 ji 29 
2 1 14 1 
2 1 1 
3 2 1 
3 2 6 
4 2 22 
4 2 18 
5 3 17 
e 3 20 
6 3 5 
6 3 2 
The input is: 
GLM 
USE SCHOOLS 


CATEGORY TEACHER, SCHOOL 


MODEL LEARNING = CONSTANT + SCHOOL « TEACHER (SCHOOL) 
ESTIMATE 
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The output is: 
Dependent Variable | LEARNING 
N i 12 
Multiple R H 0.972 
Squared Multiple R | 0.945 


Analysis of Variance 


| Type III SS df Mean Squares F-ratio p-value 
——— LC cag ace Rito Ren pos Nal ds 


e 156.500 2 78.250 11.179 0.009 
TEACHER (SCHOOL) } 567.500 3 189.167 27.024 0.001 
Error i 42.000 6 7.000 


Your data can use any codes for TEACHER, including a separate code for every teacher 
in the study, as long as each teacher within a given school has a different code. GLM 
will use the nesting specified in the MODEL statement to determine the pattern of 
nesting. You can, for example, allow teachers in different schools to share codes. 
This example is a balanced nested design. Unbalanced designs (unequal number of 
cases per cell) are handled automatically in SYSTAT because the estimation method 


is least-squares. 


Example 7 
Split Plot Designs 


The split plot design is closely related to the nested design. In the split plot, however, 
plots are often considered a random factor; therefore, you have to construct different 
error terms to test different effects. The following example involves two treatments: A 
(between plots) and B (within plots). The numbers in the cells are the YIELD of the 


crop within plots. 


Al A2 
PLOTI PLOT2 PLOT3 PLOT4 
BI 0 3 4 5 
B2 0 1 2 4 
B3 5 5 7 6 
3 4 8 6 
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Here are the data from the PLOTS data file in the form needed by SYSTAT: 


PLOT A 


YIELD 


PHP RU U U U I H2 I Hee 
NNNNNNNNK Ke eee eee 
4 oU No — oU Ho o— oU Ho — 4 U M — 
AAR tw 0 I2 RR HEU WUWUSCS 


To analyze this design, you need two different error terms. For the between-plots 

effects (4), you need "plots within 4". For the within-plots effects (B and A*B), you 

need “B by plots within 4”, | 
First, fit the saturated model with all the effects and then specify different error 1 

terms as needed. 


The input is: 


GLM 
USE PLOTS 
CATEGORY PLOT, A, B 


MODEL YIELD = CONSTANT + A + B + A*B + PLOT(A) + B*PLOT(A) 
ESTIMATE 


The output is: 


Dependent Variable | YIELD 
N i 1 


6 
Multiple R i 1.000 
Squared Multiple R ! 1.000 
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Analysis of Variance 


Source | Type III SS df Mean Squares  F-ratio p-value 
A ! 27.563 1 27.563 
B ! 42.687 3 14.229 
A*B 1 2.187 3 0.729 
PLOT (A) 3.125 2 1.563 
B*PLOT(A) | 7.375 6 1.229 
Error H 0.000 0 . 


You do not get a full ANOVA table because the model is perfectly fit. The coefficient of 
determination (squared multiple R) is 1. Now you have to use some of the effects as 
error terms. 


Between-Plots Effects 
Let's test for between-plots effects, namely A. 


The input is: 


HYPOTHESIS 
EFFECT A 
ERROR PLOT (A) 
TEST 


The output is: 
Test for effect called: A 
Test of Hypothesis 


Source | 


The between-plots effect is not significant (p-value — 0.052). 


Within-Plots Effects 


To do the within-plots effects (B and A*B), the input is: 


HYPOTHESIS 
EFFECT B 

ERROR B*PLOT (A) 
TEST 
HYPOTHESIS 
EFFECT A*B 
ERROR B*PLOT (A) 
TEST 
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The output is: 
Test for effect called: B 
Test of Hypothesis 


Source ; ss df Mean Squares F-ratio p-value 


14.229 11.576 0.007 


Test for effect called: A*B 
Test of Hypothesis 


Source | SS df Mean Squares  F-ratio p-value 
Pl A ~~ 
Al | 0.188 1 0.188 0.153 0.710 
A2 t 0.021 1 0.021 0.017 0.901 
A3 | 1.688 1 1.688 1.373 0.286 
A | 2.187 3 0.729 0.593 0.642 
Error | 7.375 6 1.229 


Here, we find a significant effect due to factor B (p-value = 0.007), but the interaction 
is not significant (p-value — 0.642). 

This analysis is the same as that for a repeated-measures design with subjects as 
PLOT, groups as A, and trials as B. Because this method becomes unwieldy for a large 
number of plots (subjects), SYSTAT offers a more compact method for repeated 
measures analysis as an alternative. 


Example 8 
Latin Square Designs 


A Latin square design imposes a pattern on treatments in a factorial design to save 
experimental effort or reduce within-cell error. As in the nested design, not all 
combinations of the square and other treatments are measured, so the model lacks 
certain interaction terms between squares and treatments. GLM can analyze these 
designs easily if an extra variable denoting the square is included in the file. The 
following fixed-effects example is from Kutner et al. (2004). The SQUARE variable is 
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represented in the cells of the design. For simplicity, the dependent variable, 
RESPONSE, has been left out. 


dayl day2 day3 day4 day5 
weekl D C A B E 


week2 C B E A D 
week3 A D B E C 
week4 E A C D B 
week5 B E D Cc A 


You would set up the data as shown below (the LATIN file). 


DAY WEEK SQUARE RESPONSE 
1 18 
17 
14 
21 
17 
13 
34 
21 
16 
15 
7 
29 
32 
27 
13 
17 
13 
24 
31 
25 


ee BLN A OK a oa Hie U DISS ROS TO eh SOP 
»uwWogmoUm»ugoum»m»Uwoum»oc 


1 
1 
1 
1 
2 
2 
2 
2, 
2 
3 
3 
3 
3 
3 
4 
4 
4 
4 
4 
5 
5 
5 
5 
5 
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To do the analysis, the input is: 


GLM 
USE LATIN 
CATEGORY DAY, WEEK, SQUARE 
MODEL RESPONSE = CONSTANT + DAY + WEEK + SQUARE 


ESTIMATE 
The output is: 
Dependent Variable | RESPONSE 
N i 25 
Multiple R i 0.931 
Squared Multiple R | 0.867 


Analysis of Variance 


Source | Type III ss df Mean Squares F-ratio p-value 


Es cd i AE rE ogi leet caet Jae lane 
DAY i 82.000 4 20.500 1.306 0.323 
WEEK i 477.200 4 119.300 7.599 0.003 
SQUARE | 664.400 4 166.100 10.580 0.001 
Error | 188.400 12 15.700 

Example 9 


Crossover and Changeover Designs 


In crossover designs, an experiment is divided into periods, and the treatment of a 
subject changes from one period to the next. Changeover studies often use designs 
similar to a Latin square. A problem with these designs is that there may be a residual 
or carry-over effect of a treatment into the following period. This can be minimized by 
extending the interval between experimental periods; however, this is not always 
feasible. Fortunately, there are methods to assess the magnitude of any carry-over 
effects that may be present. 

Two-period crossover designs can be analyzed as repeated-measures designs. More 
complicated crossover designs can also be analyzed by SYSTAT, and carry-over 
effects can be assessed. Cochran and Cox (1957) present a study of milk production by 
cows under three different feed schedules: A (roughage), B (limited grain), and C (full 
grain). The design of the study has the form of two (3 x 3 ) Latin squares: 


cow 
Latin square 1 Latin square 2 
Period I n mi IV V VI 
1 A B Cc A B Cc 
2 B C A c A B 
3 Cc A B B Cc A 
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The data are set up in the WILLIAMS data file as follows: 


COW SQUARE PERIOD FEED CARRY RESIDUAL MILK 
1 38 
25 
15 
109 
86 
39 
124 
72 
27 
86 
76 
46 


BNE U) tQ x BNR WN ox UP RN) e US T9 
e NR) U) Q) o NN WEY HSH U) € WNWH | 
VDBwBOHNOWF OH WOWNONHKS 


2 
1 
2 
2 
1 
2 
1 
1 
1 
2 
1 
2 
1 
1 
2 
2 


OQ O9 O t t t6 4 A Y U U T) I I9 ==- 
RO RO Eo B) BHO hee ee m m o Ro 


PERIOD is nested within each Latin square (the periods for cows in one square are 
unrelated to the periods in the other). The variable RESIDUAL indicates the treatment 
of the preceding period. For the first period for each cow, there is no preceding period. 
The input is: 
GLM 
USE WILLIAMS 


CATEGORY COW, PERIOD, SQUARE, RESIDUAL, CARRY, FEED 
MODEL MILK - oNSTANTsCOW+FEED+PERIOD (SQUARE) +RESIDUAL (CARRY) 


ESTIMATE 


The output is: 


Dependent Variable | MILK 
N H 18 
Multiple R | 0.995 


Squared Multiple R i 0.990 
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Analysis of Variance 


Source } Type III SS df Mean Squares  F-ratio p-value 

pera ee A EI O | ill ee 
COW i 3835.950 5 767.190 15.402 0.010 
FEED i 2854.550 2 1427.275 28.653 0.004 
PERIOD (SQUARE) | 3873.950 4 968.488 19.443 0.007 
RESIDUAL(CARRY) ; 616.194 2 308.097 6.185 0.060 
Error i 199.250 4 49.813 


There is a significant effect of feed on milk production and an insignificant residual or 
carry-over effect in this instance. 


Type I Sum-of-Squares Analysis 


To replicate the Cochran and Cox Type I sum-of-squares analysis, you must fit a new 
model to get their sum of squares. 


The input is: 


GLM 
CATEGORY COW 
MODEL MILK = CONSTANT + COW + FEED + 
PERIOD (SQUARE) +RESIDUAL (CARRY) 
ESTIMATE / SS = TYPE1 


The output is: 
Dependent Variable | MILK 
N they ere 
Multiple R | 0.995 
Squared Multiple R | 0.990 


Analysis of Variance 


Source i Mean Squares 
asara 004 S ond + 
COW i 5 1156.222 . 0.005 
FEED i 2 1138.389 22.853 0.006 
PERIOD(SQUARE) | 11489,111 4 2872.278 57.662 0.001 
RESIDUAL (CARRY) | 616.194 2 308.097 6.185 0.060 
Error i 199.250 4 49.813 
Example 10 


Missing Cells Designs (the Means Model) 


When cells are completely missing in a factorial design, parameterizing a model can 
be difficult. The full model cannot be estimated. GLM offers a means model 
parameterization so that missing cell parameters can be dropped automatically from 
the model, and hypotheses for main effects and interactions can be tested by specifying 
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cells directly. Examine Searle (1987), Hocking (1985), or Milliken and Johnson (1984) 
for more information in this area. 

Widely favored for this purpose by statisticians (Searle, 1987; Hocking, 1985; 
Milliken and Johnson, 1984), the means model allows: 
m Tests of hypotheses in missing cells designs (using Type IV sum of squares) 
m Tests of simple hypotheses (for example, within levels of other factors) 
m The use of population weights to reflect differences in subclass sizes 


Effects coding is the default for GLM. Alternatively, means models code predictors as 
cell means rather than effects, which differ from a grand mean. The constant isomitted, 
and the predictors are 1 for a case belonging to a given cell and 0 for all others. When 
cells are missing, GLM automatically excludes null columns and estimates the 
submodel. 

The categorical variables are specified in the MODEL statement differently for a 
means model than for an effects model. Here are some examples: 


MODEL Y - A*B / MEANS 


MODEL Y = GROUP*AGE*SCHOOL$ / MEANS 


These two models generate fully factorial designs (A by B and group by AGE by 
SCHOOLS). Notice that they omit the constant and main effects parameters because 
the means model does not include effects or a grand mean. Nevertheless, the number 
of parameters is the same in the two models. The following are the effects model and 
the means model, respectively, for a 2 x 3 design (two levels of A and three levels of 
B): 

MODEL Y = CONSTANT + A + B + A*B 


al bi b2 albl alb2 


A B m 

1 1 1 1 1 0 1 0 
1 2 1 1 0 1 0 1 
1 3 1 1 ES -1 -1 -1 
2 1 1 -1 1 0 —1 0 
2 2 1 E 0 1 0 -l 
2 3 1 -1 -1 -l l E 


MODEL Y = A*B / MEANS 
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A B albl  alb2  alb3  a2bl  a2b2  a2b3 
1 1 1 0 0 0 0 0 
1 2 0 1 0 0 0 0 
1 3 0 0 1 0 0 0 
2 1 0 0 0 1 0 0 
2 2 0 0 0 0 1 0 
2 3 0 0 0 0 0 1 


Means and effects models can be blended for incomplete factorials and others desi gns. 
All crossed terms (for example, A*B) will be coded with means design variables 
(provided the MEANS option is present), and the remaining terms will be coded as 
effects. The constant must be omitted, even in these cases, because it is collinear with 
the means design variables. All covariates and effects that are coded factors must 
precede the crossed factors in the MODEL statement, 

Here is an example, assuming A has four levels, B has two, and C has three. In this 
design, there are 24 possible cells, but only 12 are nonmissing. The treatment 
combinations are partially balanced across the levels of B and C. 


MODEL Y = A + B*C / MEANS 


A B c al a2 a3 biel blc2 ble3 b2cl b2c2 b2c3 
1 1 1 1 0 0 1 0 0 0 0 0 
3 1 1 0 0 1 1 0 0 0 0 0 
2 1 2 0 1 0 0 1 0 0 0 0 
4 1 2 =l -I -I 0 1 0 0 0 0 
1 1 3 1 0 0 0 0 1 0 0 0 
4 1 3 =1 -l -1 0 0 1 0 0 0 
2 2 1 0 1 0 0 0 0 1 0 0 
3 2 1 0 0 1 0 0 0 0 1 0 
2 2 2 0 1 0 0 0 0 0 I 0 
4 2 2 ~l -1 -1 0 0 0 0 1 0 
1 2 3 1 0 0 0 0 0 0 0 l 
3 2 3 0 0 1 0 0 0 0 0 1 
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Nutritional Knowledge Survey 


The following example, which uses the data file M/202, is from Milliken and Johnson 
(1984). The data are from a home economics survey experiment. DIFF is the change 
in test scores between pre-test and post-test on a nutritional knowledge questionnaire. 
GROUP classifies whether or not a subject received food stamps. AGE designates four 
age groups, and RACES was their term for designating Whites, Blacks, and Hispanics. 


Group 0 Group 1 
1 3 4 1 2 3 
w 1 3 6 9 10 13 15 
H 5 12 
B 2 4 7 8 11 14 


Empty cells denote age/race combinations for which no data were collected. Numbers 
within cells refer to cell designations in the Fisher LSD pairwise mean comparisons at 


the end of this example. 


To first fit the model, the input is: 


GLM 
USE MJ202 
CATEGORY GROUP AGE RACES 
MODEL DIFF - GROUP*AGE*RACE$ / MEANS 
ESTIMATE 


The output is: 
Means Model 
107 


Multiple R | 0.538 
Squared Multiple R | 0.289 


Dependent Variable | DIFF 
N i 


säs WARNING ***. Missing cells encountered. Tests of factors will not appear. 


Ho: All means equal. 
Unweighted Means Model 


Analysis of Variance 


Source | ss df 


Model 1068.546 14 
Error | 2627.472 92 
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We need to test the GROUP main effect. The following notation is equivalent to 
Milliken and Johnson's. Because of the missing cells, the GROUP effect must be 
computed over means that are balanced across the other factors. 

In the drawing at the beginning of this example, notice that this specification 
contrasts all the numbered cells in group 0 (except 2) with all the numbered cells in 
group 1 (except 8 and 15). 


The input is: 

HYPOTHESIS 

NOTE 'GROUP MAIN EFFECT 

SPECIFY 
GROUP[0] AGE[1] RACE$[W] + GROUP[0] AGE[2] RACE$ [W] +, 
GROUP[0] AGE[3] RACES[B] + GROUP[0] AGE[3] RACE$ [H] +, 
GROUP [0] AGE[3] RACESIW] + GROUP[0] AGE[4] RACES[B] -, 
GROUP[1] AGE[1] RACES[W] + GROUP[1] AGE[2] RACES[W] +, 
GROUP[1] AGE[3] RACES[B] « GROUP[1] AGE[3] RACE$ [H] +, 

un GROUP[1] AGE[3] RACES[W] + GROUP[1] AGE [4] RACES [B] 


1.000 


1.000 


A Matrix 


71.000 -1.000 -1.000  -1.000 0.000 
Null Hypothesis Value for D 

0.000 

Test of Hypothesis 


Source df Mean Squares F-ratio p-value 
Hypothe: 1 75.738 2.652 0.107 
Error 92 28.559 
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The computations for the AGE main effect are similar to those for the GROUP main 
effect: 


HYPOTHESIS 
NOTE 'AGE MAIN EFFECT' 
SPECIFY, 
GROUP[1] AGE[1] RACES[B] + GROUP[1] AGE[1] RACE$ [W] =, 
GROUP[1] AGE[4] RACE$[B] + GROUP [1] AGE [4] RACE$ [W] ;, 
GROUP [0] AGE[2] RACE$[B] + GROUP [1] AGE [2] RACE$ [W] =, 
GROUP [0] AGE[4] RACE$[B] + GROUP[1] AGE [4] RACE$ [W] ;, 
GROUP [0] AGE [3] RACES[B] + GROUP[1] AGE [3] RACE$ [B] +, 
GROUP [1] AGE[3] RACE$[W] =, 
GROUP [0] AGE[4] RACES[B] + GROUP [1] AGE [4] RACE$ [B] +, 
GROUP [1] AGE[4] RACES [W] 
TEST 
The output is: 
A Matrix 


+ 

1; 0. 000 

2 | 0.000 0.000 0.000  -1.000 
3 | 1.000 0.000 1.000 -1.000 -1.000 
D Matrix 

1 0.000 

2 0.000 

3 0.000 


ss df Mean Squares F-ratio 


Error 


The GROUP by AGE interaction requires more complex balancing than the main 
effects. It is derived from a subset of the means in the following specified combination. 


Again, check Milliken and Johnson to see the correspondence. 
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The input is: 

HYPOTHESIS 

NOTE 'GROUP BY AGE INTERACTION' 

SPECIFY, 
GROUP [0] AGE[1] RACES[W] - GROUP[O] AGE[3] RACES[W] -, 
GROUP[1] AGE[1] RACES[W] + GROUP[1] AGE[3] RACES[W] +, 
GROUP[0] AGE[3] RACES[B] - GROUP[O] AGE[4] RACES[B] -, 
GROUP[1] AGE[3] RACE$[B] + GROUP[1] AGE[4] RACES [B] 20.0; , 
GROUP[0] AGE[2] RACES[W] - GROUP[0] AGE[3] RACES[W] -, 
GROUP[1] AGE[2] RACES[W] + GROUP[1] AGE[3] RACES[W] +, 
GROUP[0] AGE[3] RACE$[B] - GROUP[0] AGE[4] RACES [B] -, 
GROUP [1] AGE[3] RACES[B] + GROUP[1] AGE[4] RACES [B] -0.0;, 
GROUP[0] AGE[3] RACE$[B] - GROUP[0] AGE[4] RACES [B] =; 
GROUP [1] AGE[3] RACE$[B] + GROUP[1] AGE[4] RACES[B]=0.0 

TEST 


. 1.000 0.000 
1.000 1.000 0.000 
0.000 1.000 0.000 


F-ratio p-value 
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The following commands are needed to produce the rest of Milliken and Johnson's 
results. The remaining output is not listed. 


HYPOTHESIS 


NOTE 'RACE$ MAIN EFFECT' 


SPECIFY, 
GROUP [0] 
GROUP [1] 
GROUP [1] 
GROUP [0] 
GROUP [1] 
GROUP [1] 
GROUP [0] 
GROUP [0] 

TEST 

HYPOTHESIS 


AGE[2] 
AGE[1] 
AGE [4] 
AGE [2] 
AGE[1] 
AGE [4] 
AGE [3] 
AGE [3] 


NOTE 'GROUP*RACE$' 


SPECIFY, 
GROUP [0] 
GROUP [1] 


AGE [3] 
AGE [3] 


RACES [W] =0.0;, 


GROUP [0] 
GROUP [1] 
TEST 
HYPOTHESIS 


NOTE 'AGE*RACES' 


SPECIFY, 
GROUP [1] 
GROUP [1] 


AGE [3] 
AGE [3] 


AGE [1] 
AGE [4] 


RACES [W] =0.0;, 


GROUP [0] 
GROUP [0] 


AGE [2] 
AGE [3] 


RACES [W] =0.0;, 


GROUP [1] 
GROUP [1] 
TEST 


AGE [3] 
AGE [4] 


RACES[B] + GROUP [0] 


RACES [B] + GROUP [1] 
RACE$ [B] =, 

RACE$ [W] + GROUP [0] 
RACES[W] + GROUP [1] 
RACES [W];, 

RACES [H] + GROUP [1] 
RACES [W] + GROUP [1] 
RACES[B] - GROUP [0] 
RACE$ [B] + GROUP [1] 
RACES[H] - GROUP [0] 
RACES(H] + GROUP 11] 
RACE$ [B] - GROUP [1] 
RACE$ [B] + GROUP [1] 
RACES[B] - GROUP [0] 
RACES[B] + GROUP [0] 
RACES[B] - GROUP [1] 
RACE$[B] + GROUP [1] 


Finally, Milliken and Johnson do pairwise comparisons: 


HYPOTHESIS 


POST GROUP*AGE*RACES / LSD 


TEST 


AGE [3] 
AGE [3] 


AGE [3] 
AGE [3] 


AGE [3] 
AGE [3] 


AGE [3] 
AGE [3] 


AGE [3] 
AGE [3] 


AGE [1] 
AGE [4] 


AGE [2] 
AGE [3] 


AGE [3] 
AGE [4] 


RACES [B] 
RACES [B] 


RACES [W] 
RACES [W] 


RACES[H] =, 


RACES [W] 


RACES[W] -, 


RACES[W] -, 


RACES [W] «0.0 


RACES[W] -, 


RACES [W] -, 


RACES [W] -, 
RACES [W] =0.0 
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The following is the matrix of comparisons printed by GLM. The matrix of mean 
differences has been omitted. 


Post Hoc Test of DIFF 
Using unweighted means. 
Using model MSE of 28.559 with 92 df. 


Fisher's Least-Significant-Difference Test 


GROUP (i) *AGE (i- GROUP (j) *AGE (j- 
) *RACES (i) ) *RACES (3) Difference p-value 95.0$ Confidence Interval 
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0*3*H 
0*3*H 
0*3*H 
0*3*H 
0*3*W 
0*3*W 
Q*3*W 
0*3*wW 
0*3*W 
0*3*W 
0*3*w 
0*3*W 
0*3*w 
0*4*B 
0*4*B 
0*4*B 
0*4*B 
0*4*B 
0*4*B 
0*4*B 
0*4*B 
1*1*B 
1*1*B 
1*1*B 
AID 
1*1*B 
Trp 
1*1*B 
1*1*W 
1*1*W 
1*1*W 
1*1*W 
1*1*W 
1*1*W 
1*2*W 
1*2*W 
1*2*W 
1*2*W 
1*2*W 
1*3*B 
1*3*B 
1*3*B 
1*3*B 
1*3*B 
1*3*H 
Sp 
1*3*W 
1*3*W 
1*4*B 


* This test controls the comparisonwise error rate but not the family-wise error rate. 


Within group 0 (cells 1-7), there are no significant pairwise differences in the average 
test score changes. The same is true within group 1 (cells 8-15). 
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Example 11 
Covariance Alternatives to Repeated Measures 


Analysis of covariance offers an alternative to repeated measures in a pre-post design. 
You can use the pre-test as a covariate in predicting the post-test. This example shows 
how to do a two-group, pre-post design: 
GLM 
USE FILENAME 
CATEGORY GROUP 


MODEL POST = CONSTANT + GROUP + PRE 
ESTIMATE 


When using this design, be sure to check the homogeneity of slopes assumption. Use 
the following commands to check that the interaction term, GROUP*PRE, is not 
significant: 
GLM 
USE FILENAME 
CATEGORY GROUP 


MODEL POST = CONSTANT + GROUP + PRE + GROUP*PRE 
ESTIMATE 


Example 12 
Weighting Means 


Sometimes you want to weight the cell means when you test hypotheses in ANOVA. 
Suppose you have an experiment in which a few rats died before its completion. You 
do not want the hypotheses tested to depend upon the differences in cell sizes (which 
are presumably random). Here is an example from Morrison (2004). The data 
(MOTHERS) are hypothetical profiles on three scales of mothers in each of four 
socioeconomic classes. 

Morrison analyzes these data with the multivariate profile model for repeated 
measures. Because the hypothesis of parallel profiles across classes is not rejected, you 
can test whether the profiles are level. That is, do the scales differ when we pool the 
classes together? 

Pooling unequal classes can be done by weighting each according to sample size or 
averaging the means of the subclasses. First, let’s look at the model and test the 
hypothesis of equality of scale parameters without weighting the cell means. 
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The input is: 


GLM 
USE MOTHERS 


MODEL SCALE(1) SCALE(2) SCALE(3) = CONSTANT+CLASS 


CATEGORY CLASS / EFFECT 
ESTIMATE 

HYPOTHESIS 

EFFECT CONSTANT 

CMATRIX [ 1 -1 0;0 1 -1 ] 
TEST 


The output is: 
Dependent Variable Means 
SCALE(1)  SCALE(2)  SCALE(3) 


Estimates of Effects B = (xx Y 
SCALE (2) SCALE 


SCALE (1) 


+ 
CONSTANT | 13.700 14.550 
CLASS 11 4.300 5.450 4.763 
CLASS £2 0.100 0.650 -0.787 
CLASS Hx -0.700 -0.550 0.012 


Test for effect called: CONSTANT 


C Matrix 


.000 .000 0.000 


o + 


1 14.01 1 14.012 
Error | 51.200 17 3.012 
2 i 3.712 1 3.712 1.026 
Error | 61.500 17 3.618 


Multivariate Test Statistics 


Statistic df 
Wilks's Lambda e +e 
Pillai Tra r 

i race 2, 16 


Hotelling-Lawley Trace 


Notice that the dependent variable means differ from the CONSTANT. The CONS TANT 


in this case is a mean of the cell means rather than the mean of all the cases. 
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Weighting by the Sample Size 


Suppose you believe (as Morrison does) that the differences in cell sizes reflect 
population subclass proportions, then you need to weight the cell means to get a grand 
mean; for example: 


8(11) + S(t) + 413) + 4(u4) 
Expressed in terms of our analysis of variance parameterization, this is: 
S(u + a4) + S( + ay) + 4(u + a3) + 4(u +04) 


Because the sum of effects is 0 for a classification and because you do not have an 
independent estimate of CLASS4, this expression is equivalent to: 


8(1 + 04) + 5(u + a5) + 4(u + 03) + 4(1 - à - 05 - 03) 
which works out to: 

21p + 4(a4) + 1(a5) + 0(a3) 
Use AMATRIX to test this hypothesis. 


The input is: 


HYPOTHESIS 

AMATRIX [21 4 1 0] 
CMATRIX [1 -1 0; 0 1 -1] 
TEST 


The output is: 


A Matrix 


21.000 4.000 1.000 0.000 


C Matrix 


.000 
1.000 -1.000 
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Univariate F Tests 


Source | Type III SS df Mean Squares 
+ 

1 25.190 1 25.190 
Error 51.200 17 3.012 
2 1.190 e 1.190 
Error 61.500 17 3.618 
Multivariate Test Statistics 

Statistic | Value F-ratio 
A a E re 
Wilks's Lambda 0.501 7.959 
Pillai Trace | 0.499 7.959 
Hotelling-Lawley Trace | 0.995 7.959 
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F-ratio 


8.364 
0.329 


This is the multivariate F-ratio statistic that Morrison gets. For these data, we prefer 
the weighted means analysis because these differences in cell frequencies probably 
reflect population base rates. They are not random. 


Example 13 
Hotelling's T-Square 


You can use GLM to calculate Hotelling's T-square statistic. 


One-Sample Test 


For example, to get a one-sample test for the variables X and 


dependent variables. 


The input is: 


GLM 
USE FILENAME 
MODEL X, Y - CONSTANT 
ESTIMATE 


Y, select both X and Y as 


The F-test for CONSTANT is the statistic you want. It is the same as the Hotelling's T? 


for the hypothesis that the population 


means for X and Y are 0. 


You can also test against the hypothesis that the means of X and Y have particular 
nonzero values (for example, 10 and 15) by using: 


HYPOTHESIS 
DMATRIX [10 15] 
TEST 
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Two-Sample Test 


For a two-sample test, you must provide a categorical independent variable that 
represents the two groups. 


The input is: 


GLM 
CATEGORY GROUP 
MODEL X,Y = CONSTANT + GROUP 
ESTIMATE 


Example 14 
Discriminant Analysis 


This example uses the /R/S data file. Fisher used these data to illustrate his 
discriminant function. To define the model: 
GLM 
USE IRIS 
CATEGORY SPECIES 
MODEL SEPALLEN SEPALWID PETALLEN PETALWID = CONSTANT +, 


SPECIES 
ESTIMATE 


HYPOTHESIS 
EFFECT SPECIES 
SAVE CANON 
TEST 


SYSTAT saves the canonical scores associated with the hypothesis. The scores are 
stored in subscripted variables named FACTOR. Because the effects involve a 
categorical variable, the Mahalanobis distances (named DISTANCE) and posterior 
probabilities (named PROB) are saved in the same file. These distances are computed 
in the discriminant space itself. The closer a case is to a particular group’s location in 
that space, the more likely it is that it belongs to that group. The probability of group 
membership is computed from these distances. A variable named PREDICT that 
contains the predicted group membership is also added to the file. 
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The output is: 

Dependent Variable Means 

SEPALLEN SEPALWID PETALLEN PETALWID 
oa am 3 119 


Estimates of Effects B = (X'X) !X'Y 


PETALWID 


| Level SEPALLEN SEPALWID PETALLEN 
CONSTANT ; 5.843 3.057 3.758 
SPECIES | 1 -0.837 0.371 -2.296 
SPECIES | 2 0.093 -0.287 0.502 
Test for effect called: SPECIES 
Null Hypothesis Contrast AB 
| SEPALLEN SEPALWID PETALLEN PETALWID 
EE i re eee ihe ee ccc 
it -0.837 0.371 -2.296 -0.953 
21 0.093 -0.287 0.502 0.127 


0.013 


Hypothesis Sum of Product Matrix H = B'A' (A (CX) A^) las 


PETALWID 


PETALLEN 


| SEPALLEN SEPALWID 
SEPALLEN 63.212 
SEPALWID | -19.953 11.345 
PETALLEN | 165.248 -57.240 
PETALWID | 71.279 -22.933 


Error Sum of Product Matrix G = E'E 


SEPALLEN 


SEPALWID Pi 


437.103 
186.774 80.413 
ETALLEN PETALWID 


+ 
SEPALLEN | 38.956 
SEPALWID | 13.630 16.962 
PETALLEN | 24.625 8.121 27.223 
PETALWID | 5.645 4.808 6.272 6.157 
Univariate F Tests 
Source | Type III SS df Mean Squares 


SEPALLEN 63.212 2 
Error 38.956 147 
SEPALWID 11.345 2 
Error 16.962 147 
PETALLEN 437.103 

Error 27.223 147 
PETALWID 80.413 2 
Error i 6.157 147 


Multivariate Test Statistics 


Statistic 


a eas A o + 
Wilks's Lambda i 
Pillai Trace i 
Hotelling-Lawley Trace | 


F-ratio p-value 


31.606 119.265 0.000 
0.265 
5.672 49.160 0.000 
0.11 
218.551 1180.161 0.000 
0.18: 
40.207 960.007 0.000 
0.042 
df p-value 
0.023 199.145 8, 288 0.000 
1.192 53.466 8, 290 0.000 
580.532 8, 286 0.000 
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THETA S$ M N p-value 


0.970 2 0.500 71.000 0.000 


Roots | Chi-square df 
Loc cr MI EA al 
1 through 2 ! 546.115 8 
2 through 2 | 36.530 3 


Canonical Correlations 


0.985 0.471 


Dependent Variable Canonical Coefficients Standardized 
by Conditional (within Groups) Standard Deviations 


t 1 2 
— —À P € 
SEPALLEN ; 0.427 0.012 
SEPALWID | 0.521 0.735 
PETALLEN | -0.947 -0.401 
PETALWID | -0.575 0.581 


Canonical Loadings (Correlations between Conditional 
Dependent Variables and Dependent Canonical Factors) 


SEPALLEN | 
SEPALWID į 
PETALLEN | -0.706 0.168 
PETALWID | 


Group Classification Function Coefficients 


i 2 3 


SEPALLEN | 23,544 15.698 12.446 
SEPALWID | 23.588 7.073 3.685 
PETALLEN | -16.431 5.211 12.767 
PETALWID į -17.398 6.434 21.079 


-86.308 -72.853 -104.368 


Canonical scores have been saved. 


The multivariate tests are all significant. The dependent variable canonical coefficients 
are used to produce discriminant scores. These coefficients are standardized by the 
within-groups standard deviations so you can compare their magnitude across 
variables with different scales. Because they are not raw coefficients, there is no need 
for a constant. The scores produced by these coefficients have an overall zero mean and 
a unit standard deviation within groups. 
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The group classification coefficients and constants comprise the Fisher discriminant 
functions for classifying the raw data. You can apply these coefficients to new data and 
assign each case to the group with the largest function value for that case. 


Studying Saved Results 


The CANON file that was just saved contains the canonical variable scores 
(FACTOR(1) and FACTOR(2)), the Mahalanobis distances to each group centroid 
(DISTANCE(1), DISTANCE(2), and DISTANCE(3)), the posterior probability for each 
case being assigned to each group (PROB(1), PROB(2), and PROB(3)), the predicted 
group membership (PREDICT), and the original group assignment (GROUP). 

To produce a classification table of the group assignment against the predicted 
group membership and a plot of the second canonical variable against the first, the 
input is: 

XTAB 
USE CANON 
PRINT NONE/ FREQ CHISQ 
TABULATE GROUP * PREDICT 


PLOT FACTOR (2) *FACTOR (1) /OVERLAY GROUP=GROUP COLOR=2,1,3, 
FILL=1,1,1 SYMBOL=4,8,5 


The output is: 
Counts 
GROUP (rows) by PREDICT (columns) 
1 2 3 Total 


1 50 0 0 50 
2 | 0 48 2 50 
3 vto 1 49 50 
PER de 
Total ; 50 49 51 150 


Chi-square Tests of Association for GROUP and PREDICT 


Test Statistic | value df  p-value 


* 
Pearson Chi-square | 282.593 4.000 
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FACTOR(2) 


10 


0 5 
FACTOR(1) 


However, it is much easier to use the Discriminant Analysis procedure. 


Prior Probabilities 


In this example, there were equal numbers of flowers in each group. Sometimes the 
probability of finding a case in each group is not the same across groups. To adjust the 
prior probabilities for this example, specify 0.5, 0.3, and 0.2 as the priors: 


PRIORS 0.5 0.3 0.2 


GLM uses the probabilities you specify to compute the posterior probabilities that are 
saved in the file under the variable PROB. Be sure to specify a probability for each 
level of the grouping variable. The probabilities should add up to 1. 


Example 15 
Principal Components Analysis (Within Groups) 


GLM allows you to partial out effects based on grouping variables and to factor residual 
correlations. If between-group variation is significant, the within-group structure can 
differ substantially from the total structure (ignoring the grouping variable). However, 
if you are just computing principal components on a single sample (no grouping 
variable), you can obtain more detailed output using the Factor Analysis procedure. 
The following data (USSTATES) comprise death rates by cause from nine census 
divisions of the country for that year. The divisions are in the column labeled DIV, and 
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the U.S. Post Office two-letter state abbreviations follow DIV. Other variables include 
ACCIDENT, CARDIO, CANCER, PULMONAR, PNEU FLU, DIABETES, LIVER, 
STATES, FSTROKE, MSTROKE. 

The variation in death rates between divisions in these data is substantial. Here is a 
grouped box plot of the second variable, CARDIO, by division. The other variables 


show similar regional differences. 


TATI 


IVISIONS 


Suppose you analyze these data ignoring DIVISIONS, the correlations among death 
rates would be due substantially to between-divisions differences. You might want to 
examine the pooled within-region correlations to see if the structure is different when 
divisional differences are statistically controlled. Accordingly, you will factor the 
residual correlation matrix after regressing medical variables onto an index variable 


denoting the census regions. 
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The input is: 


GLM 
USE USSTATES 
CATEGORY DIVISION 
MODEL ACCIDENT CARDIO CANCER PULMONAR PNEU_FLU, 
DIABETES LIVER FSTROKE MSTROKE = CONSTANT + DIVISION 
ESTIMATE 
HYPOTHESIS 
EFFECT DIVISION 
FACTOR ERROR 
TYPE CORR 
ROTATE 2 
TEST 


The hypothesis commands compute the principal components on the error (residual) 
correlation matrix and rotate the first two components to a varimax criterion, For other 
rotations, use the Factor Analysis procedure. 

The FACTOR options can be used with any hypothesis. Ordinarily, when you test a 
hypothesis, the matrix product INV(G)*H is factored and the latent roots of this matrix 
are used to construct the multivariate test statistic, However, you can indicate which 
matrix—the hypothesis (H) matrix or the error (G) matrix—is to be factored. By 
computing principal components on the hypothesis or error matrix separately, FACTOR 
offers a direct way to compute principal components on residuals of any linear model 
you wish to fit. You can use any A, C, and/or D matrices in the hypothesis you are 
factoring, or you can use any of the other commands that create these matrices. 


The output is: 


Principal Components Computed on the following Error Correlation Matrix 
ACCIDENT CARDIO CANCER PULMONAR PNEU_FLU 


reo tdem reum i tte ier reri ti Ca iim mre ert etm e Eamus 
ACCIDENT | 1.000 
CARDIO I 0.280 1.000 
CANCER i 0.188 0.844 1.000 
PULMONAR | 0.307 0.676 0.711 1.000 
PNEU FLU | 0.113 0.448 0.297 0.396 1.000 
DIABETES | 0.297 0.419 0.526 0.296 -0.123 
LIVER I -0.005 0.251 0.389 0.252 -0.138 
FSTROKE | 0.402 -0.202 -0.379 -0.190 -0.110 
MSTROKE | 0.495 -0.119 -0.246 -0.127 -0.071 


Principal Components Computed on the following Error Correlation Matrix 
DIABETES LIVER FSTROKE MSTROKE 


+ 
DIABETES | 1.000 
LIVER i 
FSTROKE | 


MSTROKE -0:076 -0.203 0.947 1.000 
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3.341 2.245 1.204 0.999 0.475 


Latent Roots 


0.364 0.222 0.119 0.033 


Loadings 


PNEU FLU ; 0.417 0.146 -0.842 -0.010 -0.042 
DIABETES | 0.512 0.218 0.528 -0.580 0.068 
LIVER + 0.391 -0.175 0.400 0.777 -0.044 
FSTROKE | -0.518 0.795 0.003 0.155 0.226 
MSTROKE | -0.418 0.860 0.025 0.138 0.204 


PULMONAR ! 
PNEU FLU | 
DIABETES | 
LIVER | 
FSTROKE | 
MSTROKE | 


Rotated Loadings on 


ACCIDENT | 0.457 0.682 
CARDIO | 0.906 -0.060 
CANCER } 0.909  -0.234 
PULMONAR | 0.838 -0.047 
PNEU FLU j| 0.441 -0.008 
DIABETES | 0.556 0.027 
LIVER 1 0.305 -0.300 
FSTROKE | -0.209 0.925 
MSTROKE | -0.093 0.951 


Sorted Rotated Loadings on first 2 Principal Components 
(Loadings less than 0.25 made 0) 


i 1 2 


ACCIDENT*CANCER i 0. 

CARDIO*CARDIO | 0.906 0.000 
CANCER* PULMONAR { 0.838 0.000 
PULMONAR*DIABETES | 0.556 0.000 
PNEU FLU*MSTROKE | 0.000 0.951 
DIABETES*FSTROKE į 0.000 0.925 
LIVER*ACCIDENT | 0.457 0.682 
FSTROKE*LIVER 1 0.305 -0.300 
MSTROKE*PNEU FLU | 0. 441 0.000 
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Notice the sorted, rotated loadings. When interpreting these values, do not relate the 
row numbers (1 through 9) to the variables. Instead, find the corresponding loading in 
the Rotated Loadings table. The ordering of the rotated loadings corresponds to the 
order of the model variables. 

The first component rotates to a dimension defined by CANCER, CARDIO, 
PULMONAR, and DIABETES; the second, by a dimension defined by MSTROKE and 
FSTROKE (male and female stroke rates). ACCIDENT also loads on the second factor 
but is not independent of the first. LIVER does not load highly on either factor. 


Example 16 
Canonical Correlation Analysis 


Suppose you have 10 dependent variables, MMPI(1) to MMPI(10), and 3 independent 
variables, RATER(1) to RATER(3). Enter the following commands to obtain the 
canonical correlations and dependent canonical coefficients: 


GLM 
USE DATAFILE 
MODEL MMPI(1 .. 10) = CONSTANT + RATER(1) + RATER(2) + RATER(3) 
ESTIMATE 


STANDARDIZE 
EFFECT RATER(1) € RATER(2) & RATER (3) 
TEST 


The canonical correlations are displayed; you can rotate the dependent canonical 
coefficients by using the Rotate option. 

To obtain the coefficients for the independent variables, run GLM again with the model 
reversed: 


MODEL RATER(1 .. 3) = CONSTANT + MMPI(1) + MMPI(2), 
+ MMPI(3) + MMPI(4) + MMPI(5), 
+ MMPI(6) + MMPI(7) + MMPI(8), 
+ MMPI(9) + MMPI (10) 
ESTIMATE 
HYPOTHESIS 
STANDARDIZE TOTAL 
EFFECT MMPI(1) & MMPI(2) & MMPI(3) & MMPI(4) &, MMPI(5) & 
ae MMPI(6) & MMPI(7) & MMPI(8) &, MMPI(9) & MMPI (10) 
T 
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Example 17 
Mixture Models 


Mixture models decompose the effects of mixtures of variables on a dependent 
variable. They differ from ordinary regression models because the independent 
variables sum to a constant value. The regression model, therefore, does not include a 
constant, and the regression and error sum of squares have one less degree of freedom. 
Marquardt and Snee (1974) and Diamond (2001) discuss these models and their 
estimation. 

Here is an example using the PUNCH data file from Cornell (1985). The study 
involved effects of various mixtures of watermelon, pineapple, and orange juice on 
taste ratings by judges of a fruit punch. 


The input is: 


GLM 
USE PUNCH 
MODEL TASTE = WATRMELN + PINEAPPL + ORANGE + , 
WATRMELN* PINEAPPL + WATRMELN*ORANGE + , 
PINEAPPL*ORANGE 
ESTIMATE / MIX 


The output is: 
Dependent Variable i TASTE 
N 1 18 
Multiple R 1 0.969 
Squared Multiple R 1 0.939 
Adjusted Squared Multiple R | 0.913 
Standard Error of Estimate | 0.232 


Regression Coefficients B = (x'x) ix" 
Std. 


t p-value 


WATRMELN 
PINEAPPL 
ORANGE H 
WATRMELN*PINEAPPL | 
WATRMELN*ORANGE i 
PINEAPPL*ORANGE ' 


.267 0.657 


4 

6 

7. . 
2.400 0.657 
1 . . » 
2.200 0.657 -0.293 0.667  -3.351 
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Confidence Interval for Regression Coefficients 


i 95.0% Confidence Interval 
Effect | Coefficient Lower Upper 


mom + 
WATRMELN i . 4 4 . 
PINEAPPL i 6.333 6.041 6.625 l. 
ORANGE i 7.100 6.808 7.392 1. 
WATRMELN*PINEAPPL | 2.400 0.969 3.831 1. 
WATRMELN*ORANGE $ 1.267 -0.164 2.697 1. 
PINEAPPL*ORANGE 1 -2.200 -3.631 -0.769 1. 


Analysis of Variance 


Source i Type III ss df -ratio p-value 
neppure een A da 
Regression ; 9.929 5 . 36.852 0.000 
Residual i 0.647 12 0.054 


Not using a mixture model produces a much larger R (0.999) and an F-value of 
2083.371, both of which are inappropriate for these data. Notice that the Regression 
Sum-of-Squares has five degrees of freedom instead of six as in the usual zero-intercept 
regression model. We have lost one degree of freedom because the predictors sum to 1. 


Example 18 
Partial Correlations 


Partial correlations are easy to compute with GLM. The partial correlation of two 
variables (a and b) controlling for the effects of a third (c) is the correlation between 
the residuals of each (a and 5) after each has been regressed on the third (c). You can 
therefore use GLM to compute an entire matrix of partial correlations. 

For example, to compute the matrix of partial correlations for Y/, Y2, Y3, Y4, and 
Y5, controlling for the effects of X, select Y/ through Y5 as dependent variables and X 
as the independent variable. 


The input is: 


GLM 


MODEL Y(1 .. 5) - CONSTANT « X 
PLENGTH LONG 
ESTIMATE 


Look for the Residual Correlation Matrix in the output; it is the matrix of partial 
correlations among the y’s given x. If you want to compute partial correlations for 
several x’s, just select them (also) as independent variables. 
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Computation 


Algorithms 


Centered sum of squares and cross products are accumulated using provisional 
algorithms. Linear systems, including those involved in hypothesis testing, are solved 
by using forward and reverse sweeping (Dempster, 1969). Eigensystems are solved 
with Householder tridiagonalization and implicit QL iterations. For further 
information, see Wilkinson and Reinsch (1971) or Chambers (1977). 
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5 
Introduction to Linear Mixed Models 


Amit Saxena and Arnab Chakraborty 


Linear mixed effects models are useful for analyzing data obtained from designed 
experiments, for regression analysis and for a host of other situations, where 
traditional linear models, dealing with only fixed effects, need refinement. These 
include correlated data, clustered data, dependent data and heteroscedastic data. 
SYSTAT has provision to analyze various types of linear mixed effects models: 
variance components models, hierarchical mixed models, and mixed regression. 
SYSTAT has three different main commands for analyzing linear mixed models: VC 
for variance components models, MIXED for general linear mixed effects models, and 
MIX for mixed regression. Their usages are detailed in the subsequent chapters. This 
chapter is devoted to acquaint the user with the statistical ideas and the theory behind 
linear mixed models, as well as with the terminology that SYSTAT uses. 


Mixed Models and Paired t-test 


One way to get started with linear mixed models is by considering paired t-tests as 
linear mixed model analysis. 


Illustrative case: This example is borrowed from a Netmaster online course (see the 
list of references for the URL). Here we want to compare two methods (High 
Performance Liquid Chromatography-HPLC and Near Infra Red-NIR) to ascertain 
the amount of active content in certain tablets. Suppose that we want to test if the two 
methods yield the same average content. Data have been collected by applying the 
tests to the same set of 10 tablets (e.g., by breaking each tablet into two halves, and 
applying one method to each half, assigned at random). The resulting data are shown 
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in the following table. These data are also available in a SYSTAT file named TABLET. 


Active content of tablets by two methods: 


Tablet Methods 
HPLC NIR 
1 10.4 10.1 
2 10.6 10.8 
3 10.2 10.2 
4 10.1 99 
5 10.3 11.0 
6 10.7 10.5 
7 10.3 10.2 
8 10.9 10.9 
9 10.1 10.4 
10 9.8 9.9 


A standard method to analyze this kind of data is the paired t-test. Let x; be the 
measurement by the HPLC method for the i-th tablet, and let y; be that by the NIR 
method. Then the paired t-test computes the differences 

Zi = Xi Yi 


and checks if 


2/10 


VEn -2)2/9 


is far from 0 using the t distribution with 9 degrees of freedom, where z is the mean 
of the Z;'s. 


Let us perform this test using SYSTAT . 


The input is: 


USE TABLET 
TESTING 
TTEST HPLC NIR 
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The output is: 


Paired Samples t-test on HPLC vs NIR with 10 Cases 
Alternative = ‘not * 


Mean HPLC : 10.340 
Mean NIR : 10.390 
Mean Difference : -0.050 
95.00& Confidence Interval : -0.261 to 0.161 
Standard Deviation of Difference = 0.295 
t : -0.535 
df : 9 
p-value : 0.605 


This test assumes that z/s are independently and identically distributed normal random 
variables, which is the case if, for example, each (x; ,y¡) pair is independently 
distributed as N(1;, H2, E) where X is the covariance matrix. 
However, if we consider the data set for a moment we can see that E cannot be 
justany covariance matrix. Assuming that both the methods are reasonable, it 1S highly 
likely that their measurements for the same tablet will be positively correlated. For 
instance, if a tablet has a high active content then both the measurements should be 
high. 
Popular as it is, the paired t-test nonetheless fails to take this extra information about 
the data into account. It collapses the pairs (x; ,y;) into the differences z;, and thus fails 
to utilize the correlation structure of the original data. One way to remedy this loss of 
information is to assume that each measurement is made up of three components: 
m The effect of the actual content of the tablet (which is a random variable depending 


on the manufacturing process of the tablets.) It is customary to express the effect 
ofthe i-th tablet as p + a , where p is called the mean effect, denoting the average 
level of active content that an ideal tablet is supposed to contain, while œ; denotes 
the departure of the i-th tablet from this average. 

m The effect of the measurement method (that is where our interest lies.) We shall 


denote the effect of the j-th method by p, for j=1,2. 


m Any random error, which we call £j; 


So we have the model 


yj = nte t Bit Sij 


where ¡=1,...,10 and j=1,2. Here yj; is the measurement for the i-th tablet obtained by 


the j-th method. Thus, we have renamed x; as yi] and y; as Yi 
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Readers familiar with the SYSTAT GLM command will quickly recognize this as 
a linear model. However, there is an important difference between this and the models 
fit by GLM. In GLM the parameters 1, a and f; are all (unknown) constants. But 
here the tablet effect aj s are random, since the tablets constitute a random sample 
from the population of all such tablets. The effects, u, D; are fixed as before. A linear 
model where some (or all) of the parameters are random is a called a linear mixed 
model. Here a's are the random effects, while, u f;'s are called fixed effects. We 
assume that œ; 's and £;;'s are independent Gaussian (normal) random variables with 
zero mean. SYSTAT allows various covariance structures for a, 's and ei 's. In this 
example we shall assume that a; 's distributed independently as N(0, o." ), while £; 's 
have independent N(0, o; ) distributions. It is easy to check that the correlation between 
the two measurements for the same tablet is indeed positive under this model, since 


Cov(y;¡»Y¡9) = Var(a;) = o2>0 


Let us take a look at how SYSTAT will handles this model. First, SYSTAT 
demands that the data file be organized in a way that is convenient for this computation 
as follows: 


METHODS CONTENT TABLET 


HPLC 10.4 1 
HPLC 10.6 2 
HPLC 10.2 3 
HPLC 10.1 4 
HPLC 10.3 5 
HPLC 10.7 6 
HPLC 10.3 7 
HPLC 10.9 8 
HPLC 10.1 9 
HPLC 9.8 10 
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METHODS CONTENT TABLET 


NIR 10.1 1 
NIR 10.8 2 
NIR 10.2 3 
NIR 9.9 4 
NIR 11.0 5 
NIR 10.5 6 
NIR 10,2 7 
NIR 10.9 8 
NIR 10.4 9 
NIR 9.9 10 


Using the Data=>Reshape=>Stack menu it is easy to convert the data file TABLET 
to this format, rename the columns as METHODS, Content (from the default names of 
Group$, Variable respectively) and save it as SYSTAT data file TABLET2. 


The input is: 


USE TABLET2 

ve 
CATEGORY TABLET METHODS 
MODEL CONTENT = INTERCEPT + METHODS 
RANDOM TABLET 

ESTIMATE 


Before looking at the output, we point out that our linear mixed model is of a special 
type called the variance components model, and the SYSTAT command for that is VC. 


The output is: 

Type III Tests for Fixed Effects 

Effect | Numerator df Denominator df F-ratio p-value 
aon Olan ag ew xo, 0.605 


Fixed Effects Versus Random Effects 


As pointed out in the last section, a mixed linear model is a linear model where some 
(or all) of the effects are random. These are called the random effects, while the others 
are called fixed effects. The randomness in the data is thus split up into two parts: the 
random effects and the random error. We always assume that the random errors and 

random effects are independent and are Gaussian with zero mean. The random effects 
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need not be independent among themselves. The random errors may also be 
interdependent. Owing to the presence of the random effects the original observations 
are also correlated. SYSTAT allows different covariance structures for the random 
effects as well as the random errors, as we shall see later. But first let us see why one 
would want to consider an effect in a linear model as random. 


Illustrative case: Mickey et al, (2004) discuss a data set involving two teaching 
methods and three teachers. Each teacher uses each teaching method with four 
different batches of students. The performance of each batch is measured by the 
average score of the batch in a common examination. The data set is given below: 


Comparing teaching methods 


(Scores of Students) 
TEACHER METHOD 1 METHOD 2 
1 67, 73, 59, 84 75, 61, 67, 58 
2 92, 84,94, 83 54, 78, 61, 70 
3 74,72, 76, 64 42, 44, 80 ,83 


The data are in SYSTAT file TEACH (in the format required by SYSTAT in terms of 
24 cases (rows) and three columns Score, TEACHER, and METHOD). Let Yijk denote 
the score of the k-th batch under the i-th teacher using the j-th teaching method. Then 
Jijk is the resultant of the i-th teacher effect as well as the j-th method effect. In fact, we 
can also take into account the batch effect, but for this study we shall assume that the 
batches are all more or less identical. Also, we shall ignore any interaction between 
teacher and method. (One can actually include the interaction to satisfy oneself that the 
interaction is insignificant.) So we have the linear model 


Yik = H+ Qj + B; + Eijk 


Here p is the mean effect, az, is the i-th teacher effect, and B, is the effect of the j-th 
method. The £'s, as usual, denote the random errors, 

Now let us pause for a moment and wonder why one would really collect and 
analyze a data set of this kind. In other words, what type of inference do we want to 
make? There are two possible answers to this. 

First, we may be interested in knowing how these three teachers perform using the 
two methods. This question is of interest to, for instance, the head of a school, when 
she wants to decide which method to adopt. Here she has a specific set of teachers in 
mind. 
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Second, an educator may want to compare the two teaching methods irrespective of 
the teachers. He does not have any specific set of teachers in mind. He is comparing 
the performance of method | as applied by some randomly selected teacher, with the 
performance of method 2 applied by another (possibly different) randomly selected 
teacher. 

In the first case all the effects are fixed. In the second case, the teacher effects a;’s 
are random. Let us analyze the data set under both the models to see how the inference 
differs. First, the fixed effects model. 


The input is: 


USE TEACH 
ve 

MODEL SCORE = INTERCEPT + TEACHER + METHOD 
ESTIMATE 


Notice how we have used the VC command even though we are fitting a fixed effect 
model, This is because fixed effects models are special cases of mixed effects models 
where there are no random effects. We could also have used the GLM commands. 


The input is: 


USE TEACH 
GLM 
MODEL SCORE = INTERCEPT + TEACHER + METHOD 


ESTIMATE 
Either method produces the same information. We show a relevant part of the output 
from the VC command. 


Estimates of Fixed Effects 


Standard Error 


Effect p-value 


| Estimate 


METHOD 712,417 


Type III Tests for Fixed Effects 


Source Numerator df Denominator df 


TEACHER 
METHOD | x 


Next we apply the mixed effects model where the teacher effect is random. 
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The input is: 


USE TEACH 

ve 
MODEL SCORE = INTERCEPT + METHOD 
RANDOM TEACHER 

ESTIMATE 


A relevant snippet from the output is shown below. 
Estimates of Covariance Components 


Random Effect | Description Estimate 


RR ESE o ES 
TEACHER | Variance 0.010 
| Parameter 
=e eRe ACR tuo, A 
Error variance | Variance 150.299 


| Parameter 


Estimates of Fixed Effects 


Effect | Estimate Standard Error df t p-value 


METHOD | -12.417 5.005 21  -2.481 0.022 


Type III Tests for Fixed Effects 

Effect | Numerator df Denominator df F-ratio p-value 
Se SOE A E cc SEN Ron M acum 2 
METHOD | 1 21 6.155 0.022 


Notice that the result of the analysis is now different: the p-value has gone down. This 
means that the methods appear more significantly different when used over a 
population of teachers, than when used for just a specific set of teachers. It could have 
been the other way around also. Then the interpretation would be as follows. 


m The significant difference in the fixed effects model implies that if the same teacher 
uses both the methods then the results are different. 


m The lack of significance in the mixed effects model means that a random teacher 
using one method has more or less the same performance as a (possibly different) 
random teacher using the other method. This is the case if, for instance, there is a 
lot of variability among the teachers, and the difference between the teaching 
methods is swamped out by it. A bad teacher with a good method may not perform 
much differently from a good teacher with a bad method. 
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Why Use Random Effects? 


A linear model, just like any other statistical model, tries to capture the essense of the 
process generating the data, rather than that of the data itself. We want our inference 
to hold not only for the given data set but also for future replications of the same 
experiment. So the choice of the model is dictated by what type of replications we have 
in mind. Depending on this there are different reasons behind treating an effect as 
random in a model. Here we outline three such common situations. 


m [f we plan to use the same levels of the some effect in all fresh replications, then 
we may treat the effect as fixed. However, if we plan to use fresh levels of some 
effect, then we should make the effect random. Inference based on random effects 
models are valid for a population of all possible levels of the random effects. The 
teaching method example furnished one illustration. In such situations, the random 
coefficients are all independently and identically distributed, as they represent 
randomly selected levels of the effect. So the resulting model is a variance 
components model. The next chapter has many more examples of such models in 
use for real life data sets. 

m Iin some cases, an effect may be considered random even if we plan to use the same 
levels for all future replications. Consider, for instance, a designed experiment 
where 3 operators in a factory are operating 2 machines, the response being a score 
that combines the quality and quantity of the output produced in a given amount of 
time. A suitable model for this situation may be: 


yg. = M tot Bj + Yi; * Eijk 


where y jj, is the score for the k-th run of the i-th machine operated by the j-th operator. 
If the factory has only these three operators to operate the machines, then the factory 
authorites would have to always choose the same three operators in all future 
replications of the experiment. However, the same operator may behave slightly 
differently from one replication of the experiment to the next depending on 
unpredicatable factors like his mood. In this case, we would be justified to consider the 
operator effect as random. However, since the mood variablilty of the different 
operators may be different, so here the random coefficients B;'s need not be identically 
distributed. In fact, they may also be correlated, because the moods of the all the 
operators may be affected by some common random condition prevailing during a 
replication of the experiment (e.8., weather during the experiment) that is difficult to 
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control. Indeed, McLean et al. (1991) also suggests a model where the operator effect 
is fixed, but the interaction effect ( yij ) is random. Such a model would be appropriate 
if we consider the main effect as a measure of the profficiency of the operator, which 
s not likely to change between replications. However, the mood fluctuations may affect 
how an operator operates a given machine. Such models where the random effect 
coefficents may not be independently and identically distributed are more general than 
simple variance components models. The MIXED command is designed to tackle these 
cases. Real life examples are furnished in the chapter Linear Mixed Models and 
Hierachical Linear Mixed Models. 


= A third situation that leads to random effects is where the model is developed in a 
multi-level fashion. Consider a situation where we want to linearly regress a 
response variable y on a predictor variable x. However, we believe that the 
regression slope is a random effect that depend on the values of a categorical 
variable z. Then we have a two-level model. In the f irst level we model y in terms 
of x: 


Yik 7 9 * Bj Kt in 


Here j denotes the levels of the categorical variable z. In the second level we model the 
(random) regression slope in terms of z: 


B; = ab; 


Here bj's are random effect coefficients. Putting the second level equation in the first 
we get the composite model: 


yik = At (a + b)Xis + Esa 


This means that here x is present in the fixed part ax; as well as in the random part 
Dx ¡jx. effect. If the deeper levels in a multi-level model have their own random errors, 
then they lead to random effects in the composite model. The SYSTAT specification 
for the above example is: 


MODEL Y - INTERCEPT « X 
RANDOM X / GROUP = Z 


Such models often have the same effect in both the MODEL and RANDOM lines, as x is 
in this example. Milliken and Johnson (1992) have more examples of a like nature. The 
chapter Linear Mixed Models in this manual illustrates one such model in action. 
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Some Linear Model Terminology 


Now we shall discuss some important issues about linear models, and how SYSTAT 
handles them in the context of mixed effects models. Users familiar with the GLM 
command of SYSTAT may just like to skim through this section. 


String and Numeric Variables 


A variable in SYSTAT can be either a string or numeric. The names of string variables 
end with a $ sign. The values taken by a string variable can be numeric or 
alphanumeric, but the numbers will only be interpreted as names or symbols and not 
usable for calculations. Thus string variables are used as categorical variables. 
Numeric variables take numbers as their values. However, you can ask SYSTAT to 
treat a numeric variable as categorical by using the CATEGORY command. For 
instance, the command 

CATEGORY X Y, treats x and y as categorical numeric variables. The command 

CATEGORY 


in a line by itself restores all numeric variables to their default continuous status. Thus 
string variables repersent categorial variables can be repersentedby numeric variables 
also. Numeric variables can also represent discrete or continuous variables. SYSTAT 
does not differentiate between discrete and continuous variables. 
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Estimability 


Consider the model, 


yj = Wt Qt Ej 


where i=/,2,3 and ¡=1,2. It is a well known fact from linear models theory that the 
parameters pand a; 's are not estimable from the data unless we impose further 
restrictions on the parameters. One possible restriction is to assume that sum of the d; 's 
equals 0. Then p measures the overall average, while a, 's measure the departure of 
the i-th group average from the overall average. Another popular restriction is 

a, = 0. In this case, ji measures the average of the third group, while œ; and c; 
measure the departure of the first and second group averages from that of the third 
group. 

SYSTAT calls the first restriction as effects encoding and the second restriction as 
dummy encoding. SYSTAT uses effects encoding for the fixed effects. However, as 
we shall see later, the random effects coefficients do not suffer from any estimability 
problem. So we leave them unconstrained. This is called means encoding. 


Data Layout: Multiway or Nested 


Linear models seek to explain the variation present in the data in terms of various 
effects. The part that cannot be explained thus is ascribed to random error. For instance, 
variation in the yields of a crop may be partly explained by the effects of fertilizers and 
soil type. When more than effect is present their combination determines the layout of 
the data. There are two basic layouts: multiway and nested. 

In a multiway layout all the values of each categorical variable have the same 
meaning irrespective of the values of the other categorical variables. The following is 
an example of a 2-way layout, i.e., multiway layout with just two effects. 


Illustrative case: Consider a study to investigate whether the IQ level of a person 
depends on the person's gender and lefthandedness. A typical data set will look like the 
following where y; is the IQ of the i-th person. 


Male Female 
Lefthanded — yl,y2 y3,y4 
Righthanded y5,y6 y7,y8 
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Here left-handedness means the same thing for both males and females. So in this is a 
2-way layout. 

Data sets having 2-way (or even multiway) layouts are often presented as tables like 
the above, where rows (and, ifnecessary, subrows) are devoted to some effect(s), while 
columns (and, if necessary, subcolumns) are devoted to other effect(s). Such a table 
facilitiates human reading, but SYSTAT always expects its input in a table where each 
experimental unit has its own row, and each variable has its own column. Thus, the 
above data set must be presented in the following format to SYSTAT. 


Gender$ Handedness$ IQ 


Male Left y 
Male Left y2 
Male Right ya 
Male Right Y4 
Female Left ys 
Female Left y6 
Female Right y 
Female Right ys 


Multiway layouts can be of two general types: additive and non-additive. These are 


also called models without interaction and with interaction, respectively. In SYSTAT 
interaction is called crossing. 


Illustrative case: Consider an agricultural experiment, where we want to relate the 
yields of crops to the soil type and the type of fertilizer used. Here are two possible 
hypothetical datasets.The data are in SYSTAT files AGRI and AGR2 respectively. 


A data set on agricultural yield: 


Fertilizer Soil 1 Soil 2 Soil 3 
1 10, 12 34, 30 20, 23 
2 5.4 29,28 14, 16 


Another data set on agricultural yield: 


Fertilizer Soil 1 Soil 2 Soil 3 
1 10, 12 34,30 20,23 
2 30,31 21,16 19, 25 


11-264 


Chapter 5 


Itis always a good idea to look at any data graphically before performing formal 
statistical analyses. So let us plot the two data sets as follows. We have chosen to plot 
the average yield against Soil and have used different colors for different types of 
Fertilizers. The data file AGR/ and AGR2 have been recast in the format required by 
SYSTAT and the columns have been named YIELD, FERTILIZER, and SOIL. 


The input is: 


USE AGR1 

SSAVE MEANS Y 

BY SOIL FERTILIZER 

CBSTAT YIELD / MEAN 

USE MEANS Y 

DOT YIELD*SOIL/OVERLAY GROUP-FERTILIZER LINE, 
TITLE-"Interaction chart" 


The output is: 
Interaction chart 
40, 
2 
a 
ime) 
pa 
10| 


FERTILIZER 
*1 
" x2 
05 10 15 20 25 30 35 
SOIL 
Interaction absent 


The plot shows that the lines for different fertilizers are more or less parallel. In other 
words, the general form of relation between yield and soil types is the same for all 
fertilizers. However, each fertilizer has an additive effect that shifts the yield-vs-soil 


curve up and down. This is a case where an additive model is a good choice. Here we 
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can easily compare between the fertilizers: the first is clearly better than the second. In 
SYSTAT this model will be written as: 


USE AGR1 
MIXED 

CATEGORY FERTILIZER SOIL 

MODEL YIELD = INTERCEPT + FERTILIZER + SOIL 
ESTIMATE 


Note our use of the MIXED command. Actually, we could also have used the simpler 
VC command. This input corresponds to additive model, 


Yin = pt ait Bj + Eijk 


where a's are the fertilizer effects, B;'s are the soil effects, and p is the overall mean 
effect. 


Next let us take a look at the interaction chart for the second data set. 


Interaction chart 


FERTILIZER 


La 
x2 


vs 10 15 20 25 30 35 
SOIL 
Interaction present 


Here the situation is quite different. In this case we cannot make a clear statement about 
which fertilizer is better, the first fertilizer fares well for some soil types, while the 
second fertilizer is better for the other soil types. This is manifested through the non- 
parallel nature of the two lines in the interaction chart. We say that there is interaction 
between soil type and fertilizer. This calls for a model with interaction: 
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yix = H+ o, + Bj + 75 + Eijk 


which is the same as the last model, except for the interaction effects, Y;;'s. Indeed, this 
model subsumes the additive model as a special case with Yi; 0. 


CATEGORY FERTILIZER SOIL 
MODEL YIELD = INTERCEPT + FERTILIZER + SOIL + FERTILIZER*SOIL 


ESTIMATE 


Nested Layout 


In some data sets the values of one effect A assume different meanings for different 
values of another effect B. Then we say that A is nested inside B. 


Illustrative case: Suppose that some disease can be treated by using both 
chemotherapy as well as physiotherapy. There are three possible chemotherapy 
treatments and two possible physiotherapy treatments. A typical data set will then look 
like 


Chemotherapy Physiotherapy 
Type 1 Type 2 Type 3 Type 1 Type 2 
ylll yl21 y131 y211 y221 
yll2 y122 y132 y212 y222 


This data set should be presented to SYSTAT as a table with 10 rows (one for each 
experimental unit) and 3 columns (one for therapy, one for type and one for the 
response variable.) Here the categorical variable TYPE means different things for 
Chemotherapy and Physiotherapy. We say that the TYPE effect is nested inside the 
THERAPY effect. 

Statistics textbooks often use the following model for this situation: 


yg. = H+ a; Big) + Eijk 
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where a; denotes the effect of the i-th therapy and Pg 's stand for the nested effect of 
the j-th type inside the i-th therapy. This model is written in SYSTAT syntax as follows: 


MODEL Y = INTERCEPT + THERAPY + TYPE (THERAPY) 


Balanced and Unbalanced Data 


A typical linear model data set consists of numbers that are classified into cells. In an 
agricultural study, for instance, the data may be yields of different crops, where the 
cells defined by the combinations of crop type, fertilizer type. If each cell in a cross 
model contains an equal number of observations then we call the data set as balanced. 
Such datasets are usually easier to analyze and interpret. SYSTAT allows the user to 
deal with both balanced and unbalanced data. The user does not need to explicitly 
specify whether the data set is balanced or not, since SYSTAT can figure that out from 
the data itself. In fact, one beauty of mixed models is that they provide a unified 
framework for dealing with both balanced and unbalanced data. However, there are 
certain command options that take special effect for unbalanced data sets. See the 


METHOD option for VC for details. 


SYSTAT Notation for Random Effects 


Consider the linear model, 
Yin = B+ Ai + Bi t Eijk 
where i=1,2, j=1,2 and k=1,2. This model has 5 coefficients: 1 p, 2 a 's and 2 B's. 


Here all the c;;'s are coefficients for the same effect, all the P;'s belong to another 
effect, and p is another effect. We can write this in matrix notation as: 


Y = XfP+e 
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where 
120.168 Yin Ein 
11010 Yin E112 
11001 Yi H Em 
a 
A 5 | A A £g; 
X-hoirro Y. [ys e: s 
10110 Yaz Bi 3 
10101 Yan By xi 
10101 Yan 2 
£5» 


In a mixed effects model each effect is either fixed or random. The coefficients 
corresponding to a fixed effect are treated as parameters. The coefficients for a random 
effect are assumed to be random variables that follow Gaussian distribution with mean 
Zero and some user-specified covariance Structure. For instance, we may want to treat 
the D;'s are random, keeping Hand a,,'s fixed. Then it is customary to write the fixed 
part and the random part Separately in matrix notation, as discussed below. This 
motivates the following general definition of mixed effects models. 


A linear mixed model is a linear model of the form: 
Y = XP+Zy+e 


where Y is the data vector, X and Z are known matrices (either design matrices or 
covariate matrices), B is the vector of fixed effects, y is the vector of random effects, 
and e is the random error vector. Here Y is a random vector, whose randomness comes 
partly from the random vector and partly from . We assume that 


le) 
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In particular, y and e are independent. We are denoting Var( y ) by G and Var(€ ) by R. 
Depending on the choices of X, Z, G, and R we have different types of linear mixed 
models. We shall take a brief tour through these shortly. To declare an effect as random 
we use the RANDOM command: 


MODEL Y = CONSTANT + A 
RANDOM B 


It is possible to have multiple effects in the same RANDOM line and/or multiple 
RANDOM lines in the same model. But before learning about them we need to 
understand how SYSTAT specifies the structures of G and R. 


Covariance Structures 


The model and covariance matrices for the random effects and random errors 
determine the covariance matrix of the data set as follows: 
Var(Y) = Z'GZ + R 


SYSTAT gives the user a choice from certain standard types of covariance structures 
for G and R. There are four choices for the structure of G and two choices for that of 
R. These are listed below. To illustrate these we use the following hypothetical data set 
as our running example. These data are in SYSTAT file COVSTRUCT. 


To specify the structure of the covariance matrices A and B we use the STRUCTURE 
option for the RANDOM and REPEATED commands, respectively, like this: 


USE COVSTRUCT 
MIXED 
CATEGORY P Q 
MODEL Y = INTERCEPT + P 
RANDOM Q / STRUCTURE = CS 
REPEATED / STRUCTURE = VC 
ESTIMATE 
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The values CS and VC specify the structures of the covariance matrices. These and the 
other possible structures are described below. 


wm Variance Components (VC). Here the covariance matrix is a diagonal matrix with 
equal diagonal entries, i.e., a matrix of the form o” I, where I is the identity matrix 
of appropriate size: The 4 by 4 case is shown below, 


c 000 
0000 
0000 
0000 
Notice that this structure has exactly one parameter irespective of the size of the matrix. 
Change the RANDOM line in the running example to 
RANDOM Q / STRUCTURE-VC 


to produce the output 
Estimates of Covariance Components 


Random Effect | Description Estimate 


A ananem mememe 


Q y Variance 0.001 
Parameter 

Mu TE ECT SED SER E RNC 

Error variance ; Variance 1.948 
| Parameter 


VC is the default value for the STRUCTURE option for both the RANDOM and the 
REPEATED command. Thus the above SYSTAT line could be abbreviated to just 
RANDOM Q. 


= Compound Symmetry (CS). Here the diagonal entries of the covariance matrix of 
the observations are all same, and so are the off-diagonal entries. If we write, for 
instance, 


RANDOM Q / STRUCTURE-CS 
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we get the following structure for the covariance matrix: 


ot T T-* 
To? TT 
T tot 
TOT Te 


This has two parameters irrespective of the size. In the output of the MIXED command 
the T parameter is called Compound Symmetry. 


Estimates of Covariance Components 


Random Effect Description Estimate 
Q Variance 0.000 
Parameter 


Error variance 


| Parameter 


Notice that we do not have any Compound Symmetry row for the error variance, 
because it is still using default VC covariance structure. We can of course use CS 


structure there as well: 


RANDOM Q / STRUCTURE - cs 
REPEATED / STRUCTURE = cs 


| Compound 0.560 
" 

Error variance | Variance 1.701 
| Parameter 
| Error -0.067 
| Correlation 
1 (CS) 


Observe that the t for the error covariance matrix is not printed directly. Rather it is 
divided by the error variance, to produce the error correlation. 


m Diagonal (DIAG). Here all the observations are uncorrelated but may have 
different variances. Thus, for this option the number of covariance parameters 


equals the size of the matrix. 
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o 0 0.0 
00 0 0 
0 06; 0 
0 0 00% 
If we use 


RANDOM Q / STRUCTURE=DIAGONAL 


we get the output 


Random Effect į Description Estimate 
rd A seem maint ists 
| Variance 
! 


1 
Variance 2 1.239 
Variance 3 0.000 
Variance 4 0.000 
Error variance | Variance 1.683 


Parameter 


= Unstructured (UN). This case, as the name suggests, does not put any restriction 
on the covariance matrix (except, of course, that the matrix should be positive 
definite.) 


G1) O12 915 O14 
951 97; 923 O54 
O31 93? 933 O34 
O41 042 043 O44 


If we use 


RANDOM Q / STRUCTURE-UN 
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We get the output: 


Estimates of Covariance Components 


Random Effect | 
pina epee A + 


Description 


Q | Variance 1 ? 
| Covariance (2, 0.307 
11 
| Variance 2 1.456 
| Covariance (3, 0.800 
iD 
| Covariance (3, 0.638 
12) 
| Variance 3 0.768 
1 Covariance(4, 0.868 
11) 
1 Covariance (4, 0.521 
12) 
| Covariance(4, 0.778 
13) 
Variance 4 0.814 
Error variance | Variance 1.653 


| Parameter 


Notice that only the lower triangular half of the symmetric covariance matrix is 
reported. 


Illustrative case: Consider the SYSTAT commands: 


MIXED 

CATEGORY A B 

MODEL Y = CONSTANT 

RANDOM A B A*B / STRUCTURE - VC 
ESTIMATE 


Here the random errors are uncorrelated, and so are all the random effects. The 
random errors have a common variance. The random effects of the same type have a 
common variance, which may be different from the common variance of the random 


effects of a different type. For instance, consider the model: 


yg. = HEAT Bj + Yu + Eijk 


where a, p;'s and yij's are all random effects. If we use the VC structure, then the 
£j, S are uncorrelated with a common variance O, ; Q's are uncorrelated with 
common variance o pj 'sare uncorrelated with common variance o, ; and y; 's 
are uncorrelated with common variance Gg - Thus, under this model, 


Go ee 
Var(yik) = Sa t 9b +O,+ Oe 
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More on RANDOM 


We can write in the RANDOM line any effect that can be written in the MODEL line. 
Thus, all the following are valid: 


RANDOM INTERCEPT 
RANDOM P 

RANDOM P*Q 
RANDOM P(Q) 


It is possible to list multiple effects in the same RANDOM line. Any STRUCTURE 
option in such a line applies to the multiple effects jointly. For instance 


RANDOM P Q / STRUCTURE-CS 


clubs the 2 coefficents for P and 4 coefficients for Q in a single coefficient vector of 
length 6, and postulates a 6x6 covariance matrix of the compound symmetric type. So 
here we have just two covariance parameters for all the 6 coefficients. This is easy to 
see from the following output. Estimates are all close to zero. 


Estimates of Covariance Components 


Random Effect | Description Estimate 
A A a A 
P+Q | Variance 0.000 
| Parameter 
| Compound 0.000 
| Symmetry 
——— ro mit e 
Error variance | Variance 1.948 
| Parameter 
But if we use, 


RANDOM P / STRUCTURE = CS 
RANDOM Q / STRUCTURE = 


then the covariance matrix of the 6 coefficients is block diagonal, one block for P, the 
other for Q. Each block is of the compound symmetric type, with its own parameters. 
Thus, here we have four covariance parameters in all. The relevant part of the output is: 
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Estimates of Covariance Components 


Random Effect | Description Estimate 
petisse ES e nl 
P | Variance 1.733 
| Parameter 
| Compound 0.001 


Parameter 


Compound 0.000 
Symmetry 
Error variance | Variance 1.948 


| Parameter 


Finally, SYSTAT allows block diagonal matrices where all the blocks are identical. 
Examples are 


RANDOM P / GROUP= Q STRUCTURE = cs 


Here we are nesting P inside Q. So we have 2 x 4= 8 coefficients. We are grouping the 
coefficients into four groups by the value of Q. The coefficients in different groups are 
assumed in dependent in this model. Also each group has the same covariance matrix. 
Thus the covariance matrix of all the 8 coefficients is an 8x8 block diagonal matrix 
consisting of four equal blocks of size 2, and each block has a compound symmetric 
structure. 


This is different from the model: 


STRUCTURE=CS 
or 
RANDOM P*Q / STRUCTURE = CS 


both of which have 8 coefficients, but the covariance matrices are not block diagonal, 
rather they are 8x8 compound symmetric matrices. 


It is also possible to have the GROUP option in the REPEATED command. For instance, 


REPEATED / GROUP = P 
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Using Covariates: Regression 


All our examples so far deal with categorical variables. However, SYSTAT can also 
handle the case where one or more explanatory variables are real. These correspond to 
regression situations. If only X matrix has real variables, but Z is a design matrix, then 
we have the usual regression set up. If, on the other hand, there are real variables in Z, 
we have a mixed regression situation. We illustrate these below. 


Illustrative case: First let us perform a simple linear regression analysis using MIXED. 
It is certainly an overkill to use MIXED for a simple task like this, but it makes a good 
introductory example. Consider the following hypothetical data set in the SYSTAT file 


HW. 

Height-Weight data. 

GENDER HEIGHT (Inches) WEIGHT (Kgs.) 
MALE 6.2 76 
MALE 5.8 68 
MALE 5.0 60 
MALE 5.6 58 
MALE 5.8 69 
FEMALE 5.3 70 
FEMALE 5.2 65 
FEMALE 5.5 69 
FEMALE 5.7 59 
FEMALE 5.2 62 


Initially we shall ignore GENDER, and try to fit a linear regression of WEIGHT on 


HEIGHT. 
The input is: 


USE HW 
MIXED 


MODEL WEIGHT = INTERCEPT + HEIGHT 
ESTIMATE 
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The output is: 


Analysis of Variance 


F-ratio p-value 


Source | Type III SS Numerator df Denominator df 

eee HB Cia aucune a S O ES -------------------- 
HEIGHT | 86.718 1 8.000 86.718 3,217 0.111 
ERROR | 215.682 8 26.960 


Fit Statistics 


Final L-L : -25.763 
-2L-L : 51.527 
AIC : 53.527 
AlC(Corrected) : 54.194 
BIC : 53.606 


Estimates of Variance Components 


i Variance 95.00% Confidence Interval 
Source | Components Standard Error z2 U| 
Naveen di bl o 
Error } 26.960 13.480 2.000 0.046 0.540 53.381 


Estimates of Fixed Effects 


Standard Error df 


Effect 


HEIGHT 


However, if we want to fit different lines for different genders, then we should change 
the MODEL line to, 
MODEL WEIGHT = GENDER$ + GENDERS*HEIGHT 


A relevant snippet from the output is shown below. 


Analysis of Variance 


Source | Type III SS df Denominator df Mean Squares 
per Ses area oe a rt 

GENDERS*HEIGHT 90.734 

ERROR i 211.666 


Analysis of Variance (contd...) 


Source p-value 
GENDERS*HEIGHT | 0.287 
ERROR i 


Fit Statistics 


Final L-L : -25.146 
-2L-L : 50.292 
AIC : 52.292 


AlC(Corrected) : 53.092 
BIC : 52.238 
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Estimates of Variance Components 


Variance 


dard Error 


Error | 30.238 16.163 
Estimates of Fixed Effects 


Effect 


GENDER$*HEIGHT FEMALE*HEIGHT | 


2 
1.871 


MALE*HEIGHT i 9.412 


Estimates of Fixed Effects (contd...) 


+ 
GENDERS *HEIGHT FEMALE*HEIGHT ; 0.145 

MALE* HEIGHT H 0.135 
Similarly, the model, 


p-value 


95. 


MODEL WEIGHT = INTERCEPT + GENDER$*HEIGHT 


00% Confidence Interval 


Lower 


fits two regression lines with a common intercept, while the model, 


MODEL WEIGHT = GENDER$+ HEIGHT 


fits two regression lines with a common slope. 


Uppe 


Next let us consider a model wth common slope, but where the intercept terms are 
random. The SYSTAT command lines are 


USE HW 
MIXED 


MODEL WEIGHT = INTERCEPT + HEIGHT 


RANDOM GENDERS 
ESTIMATE 
The output is: 


Estimates of Covariance Components 


Random Effect 


+ 
| Parameter 
+ 


| Parameter 


Variance 0.002 


Description Estimate 


Variance 26.960 
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Estimates of Fixed Effects 


| Estimate Standard Error 


HEIGHT i 8.569 4. 


Predictions of Random Effects 


Effect Effect Level | Esti 

GENDERS FEMALE 
MALE 

Type III Tests for Fixed Effects 


Effect | Numerator df Denominator df 


HEIGHT | 1 


Estimation and Prediction 


The covariances and the fixed effect coefficients are the parameters to be estimated in 
a linear mixed model. Besides these estimations, SYSTAT can also predict the random 
effect coefficients. SYSTAT computes Best Linear Unbiased Estimators (BLUE) for 
the fixed effects, and Best Linear Unbiased Predictors (BLUP) for random effects. 
Details are given below. SYSTAT offers a number of methods to compute the 
covariance matrices: three types of analysis of variance (ANOVA) methods, Minimum 
Variance Quadratic Unbiased Estimation (MIVQUEO), maximum likelihood method 
and Restricted or Residual Maximum Likelihood (REML) method. Among these, the 
ANOVA and MIVQUEO methods are applicable only to the models analyzed by the 
VC command. 


Estimating the Fixed Effects 


SYSTAT's estimation of the fixed effect parameters produce Best Linear Unbiased 
Estimators (BLUE), which has a number of desirable properties. By "Linear" we mean 
that the estimator scales with the input. For instance, if in a financial data set the unit 
is changed from Dollar to Euro, then the estimator will also be scaled by the Dollar- 
Euro exchange rate. The estimator is also unbiased and among all linear unbiased 
estimators it is the best in the sense that it has the minimum possible variance. 

The BLUEs and BLUPs are obtained by solving Henderson's linear Mixed Model 
Equations (MME). For justification and algorithmic details please refer to the 
Computational Details section at the end of this chapter. 
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Some salient aspects of BLUEs: 


W They are a form of Weighted Least Squares (WLS) solution. These are better than 
Ordinary Least Square (OLS) estimators if the data are correlated. 

m The weights are proportional to the precision of the estimates. Hence sometimes 
these estimators are called precision-weighted estimators. 


Predicting the Random Effects 


Next we discuss prediction of random effects. This prediction is done using the 
conditional expectation of the random effects given the observations (Y). These 
predictions are linear functions of Y and have minimum mean-square error in the class 
of all linear, unbiased predictors. Hence they are called Best Linear Unbiased 
Predictors (BLUP). The BLUPs are obtained from the solutions of Henderson's mixed 
model equations (Henderson, Kempthorne, Searle, and von Krosig , 1959). 

Some important properties of these BLUPs are as follows: 


m Under normal priors BLUPs are Empirical Bayes estimates. 


W They are a form of shrinkage estimates. Such a predictor has lower variance than 
the estimator obtained by treating the effect as fixed. 


We shall illustrate these in the Further Insights section at the end of this chapter. 


Standard Errors 


No point estimate should ever be quoted without mentioning its standard error. This 
gives an idea how different the estimate could be if we replicate the experiment . In the 
presence of random effects replicating an experiment can have more than one 
interpretation, because we have two sources of randomness: the random errors and the 
random effects. Accordingly we can have different types of replication: 


m We freshly randomize all the random effects (Broad Inference Space) 

= We hold all the random effects fixed at their levels in the original data set (Narrow 
Inference Space) 

Ww We randomize some of the random effects, and hold rest fixed at the present levels 
(Intermediate Inference Space) 

If we do not specify otherwise SYSTAT reports only the broad inference space 

standard errors of the BLUEs and BLUPs. It is possible to make SYSTAT compute 
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standard errors using narrow or intermediate inference spaces also. We shall discuss 
this in the hypotheses testing section in this chapter. 


Estimating Covariance Matrices 


As we have already touched upon, there is more than one way to estimate the 

covariance matrices using SYSTAT: 

m Analysis of variance (ANOVA) method: this has three flavors: TYPE], TYPE2 and 
TYPE3. All these are applicable only for VC models. 

m Minimum Variance Quadratic Unbiased Estimation (MIVQUEO).This is also 
applicable only for VC models. 


m Maximum Likelihood (ML) method 
m Restricted or Residual Maximum Likelihood (REML) method 


The method to be used is specified as the METHOD option for the ESTIMATE command, 
like this: 


ESTIMATE /METHOD - TYPE3 


ANOVA Method 


This is a special case of method of moments estimation, where we equate the mean sum 
of squares (MS) to their expectations, and solve the resulting system of equations. This 
method has three different versions for unbalanced data, i.e., where the numbers of 
cases in the different cells are not the same. The three versions are commonly called 
Typel, Type2 and Type3. We explain these below. 

In a variance components model we try to break the original data vector into 
different parts corresponding to the different effects and random error. Ideally the sum 
of squares (SS) of the parts should add up to the SS of the original data. This is like 
spitting the union of a number of sets into the constituent sets. In the balanced data 
situation , the constituent sets are all disjoint as in the figure below, and so the splitting 
is obvious. But if the data set is not balanced then we have overlapping sets. 
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TYPE1 


The 3 Types of Sum of Squares 


Note that in Type 1, the SS for each effect is computed after taking out the 
contributions of all other effects. In Type 3, we proceed sequentially: first A, then B 
sans the contribution of A, then A*B sans the contribution of A and B. Owing to this 
sequential nature, Type 3 SS's depend on the order in which the effects are listed. 


Type 2 is suggested as a compromise between Types 1 and 3 to achieve symmetry. 
Here, for each effect we take out the contributions of the effects of same or lower 
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orders only. For instance, the Type 2 SS for A in our example is computed as A minus 
B. The SS for B is similarly, B minus A. The interaction, being a higher order term, 
does not enter into the picture so far. To compute the SS for A*B we need to take out 
the effects of both A and B. For an example of the three types of SS in action, please 
see the Variance Components chapter in this manual. 


Here is a command line that requests the Type 1 method. 


ESTIMATE /METHOD = TYPE1 


You may replace the TYPE! keyword by TYPE2 or TYPE3 to specify the ANOVA 
estimaton method of your choice. The default is TYPE3 for models analyzed by VC. 

There is no concensus among statisticians as to which method is the best. While the 
scale tilts more toward type 3, the other two methods have their share of ardent 
supporters as well (e.g.. Milliken and Johnson (1992).) The controversy stems from the 
fact that that the different types of SS's actually test different hypotheses. The 
hypotheses tested by Types 1 and 2 involve the unequal cell frequencies or unbalanced 
cases: a fact that is vehemently disparaged by many on the ground that hypotheses 
should be statements involving only he model and not the data. Type 3 hypotheses are 
free of this blemish. However, supporters of the first two types argue that the 
dependence of a hypothesis on the cell frequencies may be justified if the cell 
frequencies reflect the underlying population sizes. Also, the hypotheses tested by the 
Type 3 method are sometimes less intuitive to interpret. 


Some salient aspects of the ANOVA methods of estimating variance components are: 
m They not dependent on the distributional assumptions on the effects. They use 
information about only the first two moments. 


m These are non-iterative methods , and usually require less computation than 
iterative methods. 


ANOVA estimates of variance components can be negative. 
= ANOVA methods in SYSTAT are applicable only to variance components models 
analyzed by VC. 


MIVQUEO 


MIVQUEO was originally proposed by Rao (1971) to estimate the variances in a 
variance components model without using any normality assumptions. This is a special 
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case of Minimum Norm Quadratic Unbiased Estimation (MINQUE) procedure. First, 
let us write the variances as a column vector, 


2 i 73 
6: (0, sea Og Ga) 


Our aim is to estimate some linear combination p's, where p is a known vector. For 
instance, if we want to estimate c," we shall take p =(0,...,0, 1)”. 


In the general MINQUE method, we start with a guess sy fors. Let Vo be the 
covariance matrix of Y using the guess so in place of s. Then we look for a quadratic 
function Y'QY of the data Y to minimize: 


trace[(QVo)?] 
with respect to Q such that 
m Q-Q' 
m QX=0 (for translation invariance) 
W pi = trace(QZiZi') (for unbiasedness) 
In MIVQUEO we take sp=(0,...,0,1) 
The SYSTAT syntax for MIVQUEO is: 


ESTIMATE /METHOD - MIVQUEO 


Some salient aspects of the MIVQUEO method of estimating variance components are: 
W MIVQUEO is applicable only for variance components models. 
m MIVQUEO is a non-iterative method. 


W MIVQUEO may produce negative variance component estimates. This is a general 
problem with method of moments estimators. They may produce estimates falling 
outside the parameter space.. 


Maximum Likelihood (ML) 


To request ML estimation in SYSTAT, the input is: 


ESTIMATE /METHOD - ML 
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Some salient aspects of the ML method of estimating covariance parameters are: 


m These estimators are consistent, asymptotically efficient, and asymptotically 
normal under quite general assumptions 

m Asymptotic covariance matrix of the estimators is produced as a by product. It is 
the inverse of the information matrix. 


SYSTAT implements ML estimation using an iterative algorithm. 
ML estimation in SYSTAT depends on normality assumptions. 
ML estimators are usually biased. 


Restricted or Residual Maximum Likelihood (REML) 


While likelihood based methods have mahy charming properties (e.g., asymptotic 
normality, achieving lowest possible variance asymptotically), ML estimators suffer 
from one drawback: they are biased estimators in general. For mixed effects models 
the origin of this bias may be explained as follows. The ML method first estimates the 
fixed effects, and then then estimates the covariance prameters by treating the residulas 
as fresh data. However, the residuals are actually more correlated than fresh data. The 
ML method, however, fails to adjust for this fact. As a result it uses higher degrees of 
freedom than what it should. The REML method is an adjustment to ML to incorporate 
a correction of the degrees of, "freedom.The Further Insights section at the end of this 
chapter provides further informationon this theme. REML, like ML, belongs to the 
family of likelihood-based methods, and hence inherits the good qualities of the family. 
Thanks to the degrees of freedom correction, REML estimators are also unbiased. So it 
is the most popular estimation method for mixed effects models. This is also the default 
in the MIXED command. 

To request this method of estimation use the following option in the ESTIMATE 


command line: 


ESTIMATE /METHOD = REML 


Some salient aspects of the REML method of estimating variance components are: 

m These estimators are consistent, asymptotically efficient, and asymptotically 
normal under quite general assumptions. 

= Asymptotic covariance matrix of the estimators is produced as a by product. It is 
the inverse of the information matrix. 

m SYSTAT implements REML estimation using an iterative algorithm. 
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m REML estimation in SYSTAT depends on normality assumptions. 
m REML estimators are unbiased. 


Testing Hypotheses 


SYSTAT can perform three different types of hypothesis tests. First, it tests each of the 
fixed effect coefficients for significance using the standard t-test. The square of the t- 
test statistic may be considered as the test statistic in an ANOVA-like F-test with a 
single degree of freedom in the numerator. SYSTAT reports the value of the t-statistic 
and also the two-sided p-value. Thus, if ar is a fixed effect coefficient, then SYSTAT 
tests 


Ho: aq; = 0 
against 
Hy: a,=0. 


Since t-distributons are symmetric, the information can be also used for one-sided 
alternatives. For one-sided alternatives, the p-value is half of the reported two-sided p- 
value. The smaller the p-value, the more significant is the coefficient. To test at a given 
level of significance, say 5%, one should reject the null hypothesis if the p-value falls 
below 0.05. 

SYSTAT also lets the user test various contrasts in three inference spaces: broad, 
intermediate and narrow, For a detailed real life example of hypothesis testing in 
different inference spaces, please see the chapter Linear Mixed Models in this manual. 
In this section we present only the conceptual underpinnings. 

Any statistical test of hypothesis looks at the data only through the test statistic, and 
tries to calibrate the observed value of the statistic as large or small. This calibration 
is achieved by comparing the observed value with values obtained from (hypothetical) 
replications from a model where the null hypothesis is indeed true. As we have already 
mentioned in the context of standard errors, there are three different replication modes 
possible for a mixed model: 


m We freshly randomize all the random effects (Broad Inference Space) 
m We hold all the random effects fixed at the same levels (Narrow Inference Space) 


Ww We randomize some of the random effects, and hold rest fixed (Intermediate 
Inference Space) 
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The hypothesis as well as the inference space are specified to SYSTAT by using three 
matrices F, R and D. Actually, SYSTAT tests the hypothesis 
Ho: FB * Ry =D. 


Notice that this equation imposes a size restriction on the matrices F, R, and D. They 
must all have the same number of rows. Also, D must be a column vector. The number 
of columns in F must equal the number of fixed effect coeffcients, while that of R must 
match the number of fixed effect coeffcients. We introduce the three matrices one by 
one. 


The F Matrix 


Consider the model, 


yg. = H+ a; + B; + Eijk 


where pand a; s are fixed effects and p, 's are random. Let us assume that i=1,2,3 
and j=1,2,3,4. Consider all the fixed effect parameters to be laid out in a row: 


H, OL], Op, 03 


To test the null hypothesis 


Ho: A; = 0 
we rewrite it as: 
0xp+1x a,+Cl* a,+0 x a; =0. 


Collecting the coefficients of the fixed effects parameters, we get the row vector (0, 1, 
-1, 0) as the F matrix. In SYSTAT we use 
HYPOTHESIS 


FMATRIX [0 1 -1 0] 
TEST 
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Similarly, the F matrix (0, 1, 0, -1) tests the equality of 1 and 3. If we stack these two 
rows one on top of the other we get 


0T-1 0 
01 0-1 


Then we are testing equality of all the a; 's. However, note that SYSTAT expects the 
F matrix to have independent rows. In other words, you cannot have redundant 
conditions on the parameters. In particular, to test equality of the œ; 's we cannot use 
the F matrix 


01-10 
01 0 -i 
00 1 —1 


Here the three rows correspond to the three conditions: a, — ,=0, a — o -0, 
A, — 04 =0, of which the last condition is redundant, since it is implied by the first two. 


The D Matrix 


Consider testing the null hypothesis œ; — ot; =2. Here we have a nonzero right hand 
side of the equation. SYSTAT calls the right hand side the D matrix of the hypothesis. 
So we shall write 
HYPOTHESIS 
FMATRIX [0 1 -1 0] 


DMATRIX [2] 
TEST 


Obviously, the F and the D matrix must have the same number of rows. 


The F matrix and the D matrix will suffice for most purposes. The resulting tests are 
performed in the so called broad inference space. To test hypotheses in the narrow or 
intermediate inference spaces we need the R-matrix, which is discussed next. 
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The R Matrix 


In a mixed model there are are two sets of random variables, the random effects and 
the random errors. If we pick some of the random effects and condition the model on 
them, then the resulting inferences are said to be done in intermediate inference space. 
If we condition our inference on all the random effects, then we are in the narrow 
inference space. 


Illustrative case: Consider the model 


yj = uta * Bj * Ej 
where ij=1,2,3. We shall treat pand o,;'s as fixed effects, and. ;'s as random. 
A typical narrow inference space hypothesis here is 


Ho: a, - 30, + 2p, + 2B, * 2B; = 0. 

Note that the hypothesis involves te random coefficients. The interpretation of this as 
follows: The test statistic here is to be calibrated against replications where the random 
effects are randomized under the constraint of Hy. Thus, we are testing the equality of 
a, and a, holding the average contribution of the ; 's fixed. This corresponds to the 
F matrix (0 3 -3 0), R matrix (2 2 2), and D matrix 0. 


The input is: 


HYPOTHESIS 
FMATRIX [0 3 -3 0] 
RMATRIX [2 2 2] 
TEST 


We have no mentioned the D matrix explicitly. So it would default to the zero matrix. 

If we keep some zero entries in R, then the corresponding random coefficients are 
uncontrained, and we have an example of intermediate inference space. (McLean, 
Sanders and Stroup 1991). SYSTAT insists that all the rows of the combined matrix [F 
R] should be linearly indepednent. D should have only one column.All the three 
matrices default to 0. 
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Pairwise Comparison Tests 


Consider the model 
yj = p tQ; t £j 


where i=1,2,3,4, and j=1,...,10. Suppose that we have rejected the null hypothesis that 
a; 's are the same. This only tells us that possibly not all the a; 's are the same, but does 
not shed any light on exactly which pair(s) of œ; 's is (are) different. In this context, 
several (multiple) comparisons or post hoc tests (say, all pairwise comparisons) are 
carried out. To guard against chance conclusions of significant differences, levels of 
significance are adjusted when multiple comparisons are made. There are a number of 
methods available to make such adjustments of the individual tests in order to achieve 
a required overall level of significance. SYSTAT implements 6 of these: 


m Bonferroni (BONF) 

Fisher's LSD (LSD) 

Tukey (TUKEY) 

Sidak (SIDAK) 

Scheffe (SCHEFFE) 

GT2 (GT2) 

For more information on multiple comparisons see Chapterl: Linear Models, 


“Pairwise Mean Comparisons" in Statistics II. The following command lines perform 
Bonferroni adjustment when all pairwise comparisons are made. 
HYPOTHESIS 


PAIRWISE X / BONF 
TEST 


Diagnostics 


Any statistical analysis makes some assumptions (in the form of a model) about the 
process that has generated the data to be analyzed. The model is at best an 
approximation to the real process. The result of the analysis is meaningful only if the 


W model assumptions are correct 


m model captures most of the important aspects of the data 
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m data does not have influential outliers (i.e., some observations that do not conform 
to the general pattern laid out by the model, and has the potential to distort the 
results.) 


Any proper statistical analysis therefore must watch out for possible violations of the 
model assumptions. SYSTAT provides a number of ways to perform such diagnostic 
checks. These come in two flavors: 


m Residual diagnostics 
m Model selection 


Residual Diagnostics 


Most statistical models aim to explain the variability present in the data set by ascribing 
parts of it to various known causes. However, it is never possible to explain the entire 
variability in this way. The remaining variability is ascribed to chance. That is why we 
have the random error terms in (mixed) models. In these random errors we sum up all 
the causes of variability that we are ignorant about. This ignorance often takes the form 
of the assumption that the errors present in the different observations are independent 
and identically distributed. To perform tests of hypotheses and confidence interval 
estimation we also assume that these random errors follow a normal distribution with 
mean 0. Residual diagnostics try to check for departures from these assumptions about 
the random error. 

The main idea behind these is essentially this: first fit the model of your choice to 
the data set, and obtain an approximation to the random errors by subtracting the 
prediction from the actual observations. These approximations are called residuals and 
are used as a proxy to the actual unobservable random errors. A word of caution, 
though: the residuals are not the actual random errors. In particular, the residuals are 
correlated even for a model with independent random errors. Residual diagnostic 
methods, therefore, should only be used as a rough check, rather than a rigorous one. 
However, residual checks should always be done. 

In a mixed model there are two different types of residuals. 


m Marginal residuals (MRESIDUALS) 
m Conditional residuals (CRESIDUALS) 


The names in parentheses are the terms used by SYSTAT. Marginal residuals are 
obtained by subtracting from the original data the prediction based on only the fixed 
effects. We can think of this prediction as over the population of levels of random 
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effects. For instance, consider the model, 


yix = Uta; + Bj £i 


where fj's are random effects. Then the marginal residuals are: 


yi - (fH Gi) 


If we include the random effects in our prediction then we get the conditional residuals 


yyk- (Å -di + B) 


You may think of the term inside parentheses as the prediction of y;jy over the specific 
levels of random effects. Ideally both these residuals should be small and show no 
pattern. To check this, one should plot these against cases, and also against the 
covariates. The variability in the conditional residual plot will be less than that in the 
marginal residual plot. 


Model Selection Criteria 


In SYSTAT, likelihood-based model selection criteria are provided for model 
selection; they are: 

m AIC =-2 Log-likelihood+2k 

m AIC (corrected) = -2 Log-likelihood+2k+2k (k+1)/ (n-k-1) 

m BIC =-2 Log-likelihood + k log (n) 

where n is the number of observations and k is the number of parameters (fixed effects, 
variance covariance parameters) estimated. In the REML method of estimation, AIC 
should be used only to compare models with the same fixed effects part. For more 
information on AIC and BIC see Chapter 2: Linear Models, “Variables selection" in 
Statistics-II. 


11-293 


Introduction to Linear Mixed Models 


Missing Observations 


There are many real-life situations where some of the observations are missing from 
the data set. However, such incomplete data sets may be quite informative if analyzed 
properly. The first thing to keep in mind when dealing with missing data is the missing 
data mechanism: "Why are the missing observations missing?" Sometimes the 
observations may be missing completely at random (MCAR). For instance, some 
observations may get lost or corrupted during transcription. This is a random process 
independent of the process of interest. In some situations, on the other hand, the 
missing process is dependent on the data. If your data consist of the measured 
intensities of stars then the observations of the weaker or more distant stars are more 
likely to be missing. This is an example of censored data. Yet another situation is 
truncation. Suppose that we are observing the lifetime of electric bulbs. The bulbs are 
turned on at the start of the experiment, and the time when they burn out are observed. 
However, the experiment is conducted within a limited amount of time. We cannot 
report the exact lifetimes of the bulbs that continue to burn after this period is over. This 
is another type of missing data mechanism. The MCAR mechanism is the most 
popularly used assumption to cope with cases where no information is available. 
SYSTAT also uses this assumption. 

When using likelihood-based methods like ML or REML, SYSTAT uses the 
Expectation-Maximization (EM) Algorithm (Dempster, Laird, and Rubin (1977)). This 
algorithm, as its name suggests, consists of two parts: the Expectation part and the 
Maximization part. In the expectation part we first pretend that we have all the 
observations, and compute the log-likelihood accordingly. This leads to a function of 
the parameters as well as the data (both observed and unobserved). Then we take 


conditional expectation of this given the observed data with respect to the distribution 


of the missing values treated as random variables. This completes the Expectation part. 
In the Maximization part, we maximize this conditional expectation with respect to the 


parameters. This process is repeated until convergence. 
Further Insights 
Henderson's Mixed Model Equation 


SYSTAT computes the BLUEs of the fixed effects and the BLUPs of the random 
effects by solving Henderson's Mixed Model Equation (MME): 
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XR'x XxXR'z ||B ERY 

ZR'X ZRUZ+G" ly DRY 
Henderson (1953) proved that the solution to this equation maximizes the joint density 
of y and y. Notice the G7! in the lower right block of the coefficient matrix. Without 
this term the solution would be just the maximum likelihood estimator considering as 
fixed effects. Also note that in the absence of any random effects term (no Z and G) the 
MME reduces to the familiar system of normal equations for linear models. 


The MME involves the covariance matrices G and R, which are unknown. So before 
we can solve the MME we need to estimate these by ML or REML. 


Some Properties of BLUPs 


We have mentioned earlier in this chapter that BLUPs are empirical Bayes estimators 
under normality assumption. Also they are a form of shrinkage estimator. We 
demonstrate these using the following example. 


Consider the model, 


Yi = pte 


where ¡=1,...,n. We shall treat as a random effect with a N(0, 1”) distribution, while 
the errors have independent N(0,o* ) distributions. This can be thought of as a 
Bayesian inference problem where the parameter has a N(0, t°) prior. Then the Bayes 
estimator of 1 will be the posterior mean (which is the conditional expectation of jt 
given the data). 


However, this is not an honest Bayes estimator as it involves the unknown variances. 
So we need to estimate them separately and plug them into the formula, producing an 
empirical Bayes estimator. The above formula for jt is precisely what one would get 
by solving the MME, which now takes the following form: 


(1',(0°D1, +74 = 1406 Dy 
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where 1, denotes the n-dimensional column vector of ones. Notice that ù is not 
unbiased if you consider it as an estimator of the fixed effect . But as a predictor of the 
random effect p it is unbiased. Finally, if we consider p as a fixed effect, then its 
BLUE is just y . Thanks to the positive term o ^x? in the denominator of the BLUP, 
the latter is always smaller in absolute value than y . In other words, we can think of 
the BLUP as a shrunk version of the BLUE. This also implies that the BLUP has lower 


variance than the corresponding BLUE. 


Why Random Effect Coefficients are Always Estimable 


SYSTAT uses effects encoding for the fixed effects, but means encoding for the 
random effects. The need for this difference stems from the fact that, unlike the fixed 
effects, the random effect coefficients are always estimable (or predictable, rather!). 
The following example demonstrates this point. 


Consider the model 


yi = Bit bet &i 


where i=1,...,10. Let us assume that the g;' ‘s are independent N(0, o!) random 
variables. Now, if the p 's are treated as fixed effects, then they are not individually 
estimable, since they enter the model only as their sum, which is estimable with BLUE 
given by y . Any two pairs of estimators ( fla, B) with sum y , would fit the data 
equally well, and there is nothing in the assumptions to choose one pair over another. 
Thus, the p's are not estimable here. If, on the other hand, the p's are random effects, 
and if we assume that they are independent N(0, 1”) random variables, then certain 
pairs will be more likely than others. Indeed, the pair where pı = H2 would be the 
most likely pair, and so the BLUP for each of the p's would be half of y . 


ML and REML 


data vector y has a multivariate normal distribution 


Under normality assumption, the 
with mean vector X . and covariance matrix V=Z'GZ + R. So the log-likelihood of 


the data is 
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Ena P" exp He -Xpv'ty- xp) 


In the ML method we maximize this with respect to B , G and R. The ML estimators 
have asymptotic normal distribution. However, for finite sample these may be biased 
estimators. REML is very similar to ML except that the estimators are unbiased. Here 
we first fit a model containing only the fixed effects. Treat the residuals from this fit as 
the new data to which we shall fit the random effects. The maximum likelihood 
estimator obtained from this second model is the REML estimator. The interested reader 
may find the details in Searle, Casella, and McCulloch (1992, p.251). The following 
illustration may serve to clear up the distinction between ML and REML. 


Illustrative case: Suppose that our data consist of y ¡,....Yy» Which we model as 
yi = Ht 


Here p is a fixed effect and the £;'s are independent N(0, o°). The ML estimator of 
o” is easily seen to be 


ERIS vl 
om = 2344 Y) 
i=l 
which is biased. Now let us "take out" the fixed effect estimate |i = y to get the 


residuals y;-. If we treat these as our new data, and compute the maximum likelihood 
estimator of o^, then we shall obtain the REML estimator 


^ ct T vl 
O REML = 9 ie?) 
i=! 


which is unbiased. 
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Ch apter 


Variance Components Models 


Arnab Chakraborty, Ravindra Jore, Sourov Ghosh, and K. Raghavendra Rao 


Variance Components (VC) can carry out estimation and hypothesis tests in a variance 
components model for both balanced and unbalanced data. A variance components 
model can have any number of fixed and/or random effects, including interactions 
(crossed effects) and nestings (nested effects). Both categorical and continuous 
variables are allowed as predictor variables. Thus VC can be used to fit mixed 
regression as well as mixed ANOVA models. The models handled by VC constitute 
a subclass of those handled by MIXED, which allows more general covariance 
structures for the random effects and the random error. The subclass of models dealt 
with by VC is arguably the most frequently used type of linear mixed models. 


Statistical Background 


A variance components model is a mixed linear model of the form 


y = XB* Zt t Zp t E 


where y is the data vector, X and Z;'s are known matrices (either design matrices or 
covariate matrices), P is the vector of fixed effects, each y, is a vector of random 
effects, and e is the random error vector. Here y is a random vector, whose 
randomness comes partly from the random vector y, and partly from £. We assume 
that the random vectors y, and € have independent Gaussian distributions with zero 
mean and covariance matrices of the form 


Var(e) = ool, Var() = ol 


11-299 


11-300 


Chapter 6 


where I is an identity matrix of appropriate order. Here each y, consists of the random 
coefficients for one random effect. The variance of the distribution may be different for 
the different effects. (SYSTAT provides the option to specify a common variance 
parameter for multiple effects.) Over and above the usual estimation techniques for 
general linear mixed models, VC offers some extra estimation techniques specially 
applicable for this subclass. Unlike the general methods (ML and REML) the special 
methods (MIVQUEO and ANOVA: TYPEI, TYPE2, and TYPE3) are non-iterative 
and require less computation. For details of these methods, please refer to chapter on 
“Introduction to Linear Mixed Models" on page 251, Statistics II. 

SYSTAT reports the BLUE's of the fixed effects and BLUP's of the random effects, 
as well as estimates of the variance parameters. Each estimation or prediction is 
accompanied with its standard error, two-sided 95% confidence interval, and a 
significance test. 

For each model you fit in VC, SYSTAT reports log-likelihood (even if you are not 
using a likelihood-based estimation method like ML or REML), Akaike Information 
Criterion (AIC), Bayes Information Criterion (BIC), and Akaike Information Criterion 
Corrected (AICc) for assessing the fit of the model. 
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Variance Components in SYSTAT 


Model Estimation (in VC) 


To fita Variance Components model, from the menus choose: 


Analyze 
Mixed Models 
Variance Components... 


Analyze: Mixed Models Variance Components 


este o 

Available variablels} Dependent 
SCORE FAR) | «Required» 

| MACHINE di tale h 

OPERATOR Se edat 

TIME 


Random effect(s): 


<~ Remove 


[V] Intercept 
DSave: — Marginal esate = iN 


DARE 


thod to estimate/predict the effects 


mponents model and the me 
ts, use the available options. 


To specify a variance Co! 
and variance componen! 
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Dependent. Dependent is the variable you want to model. Dependent variable should 
be a continuous numeric variable. 


Fixed effect(s). Select one or more continuous or categorical (grouping) variables 
which you treat as fixed effects. Fixed effects that are not denoted as categorical are 
considered covariates. If you want crossed or nested effects in your model, you need 
to build these components using Cross and Nest buttons. 


Random effect(s). Select one or more continuous or categorical (grouping) variables 
which you treat as random effects. Random effects that are not denoted as categorical 
are considered covariates. If you want interactions or nested effects in your model, you 
need to build these components using Cross and Nest buttons. 


Intercept. Includes a constant term in the model. Deselect it to obtain a model through 
the origin. 

Estimation method. Choose one among the available methods to estimate variance 
components: 


m REML. Uses Restricted Maximum Likelihood (REML) method to estimate 
covariance components. It is the default method when your model contains random 
effects. 


ML. Uses Maximum Likelihood (ML) method to estimate covariance components. 


= MIVQUE(0). Uses Minimum Variance Quadratic Unbiased Estimation method to 
estimate variance components. 


m ANOVA. To estimate variance components using ANOVA method choose 
appropriate type of sum of squares. It is the default method for fixed effects model. 


Save. Check the save option to save residuals and other data to a new data file. The 
following alternatives are available: 


m Marginal residuals. Saves marginal residuals and marginal predicted values. 


= Conditional residuals. Saves conditional residuals and conditional predicted 
values. 


Fixed effect estimates. Saves the estimates of the fixed effects. 
Random effect predictions. Saves the predictions of the random effects. 
Variance components. Saves the estimated variance components. 


Standard errors of fixed effects. Saves standard errors of the fixed effect estimates. 


Residuals/data. Saves marginal and conditional residuals along with all the 
variables in the working data file. 
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m Model. Saves marginal residuals, response variable, and the design matrices. 


Category 


To specify categorical variables, click the Category tab. Select at least one fixed or 
random effect in Model tab other than intercept to activate this tab. 


Analyze: Mixed Models Variance Components 


[ Model| Category | Options —— 


Available variable(s) 
MACHINE | 
OPERATOR 


| Remove 


nid 


rate category for cases with a missing value for the 


Missing values. Includes a sepa 
selected variable(s). 
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Options 


Use Options tab to specify computational controls for ML or REML method of 
estimation. 


Analyze: Mixed Models: Variance Components 


ML/REML 
Initial values: [ 


Type of convergence: Hernan 
Convergence criterion 


*) Relative 
Convergence: 

Number of Newton iterations: 
Number of EM iterations: 
Step-halvings: 

Estimation options 


ML/REML. The following options are available for controlling estimation using 
ML/REML methods: 


" Initial values. Use this option to provide initial values for variance components. 
Specify values for each component in the order the effects appear in your model. 
Separate the values with commas or blanks. Do not specify initial values for some 
of the parameters and leave blanks for others. If you do, SYSTAT computes initial 
values for all variance components. Make sure that the initial values are positive 
and are in a reasonable range. 
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Type of convergence. Check one of the following options to check convergence. 


Three types of convergence checks are available: 

m Hessian. Uses a quadratic form gH : g where g is the gradient vector and H is the 
hessian matrix. 

m Likelihood. Uses the difference between log-likelihood at current iteration and the 
log-likelihood at last iteration. 

m Parameter. Uses maximum of absolute differences between parameter estimates at 
current iteration and parameter estimates at last iteration. 

Convergence criterion. Two criteria are available: 

m Relative. Checks relative difference for convergence. That is, convergence 
checking is done relative to log-likelihood. It is the default option. 

m Absolute. Tests convergence directly against a value specified. 


Convergence. Specify a positive number. SYSTAT stops iterations when convergence 
value is less than this number. 


Number of Newton iterations. Use this to specify maximum number of Newton- 
Rapson iterations for fitting your model. The default is 20. 


Number of EM iterations. Use this to specify maximum number of EM iterations 
before going to Newton-Raphson iterations. Sufficient number of EM iterations 
provide good initial values for Newton-Raphson iterations. The default is 5. 


Step-halvings. Use this to specify maximum number of step halvings. The default is 50. 


Estimate options. The following estimate options are available: 

m Tolerance. A check for near singularity. Use Tolerance to guard against this 
singularity problem. 

m Confidence. Specifies the confidence coefficient for testing purposes. The default 
is 0.95. 
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Hypothesis Test 


To test hypotheses, from the menus choose: 


Analyze 
Mixed Models 
Hypothesis Test... 


Analyze: Mixed Models: Hypothesis Test 


Hypothesis: [Paiwise B 
Available effects: 
MACHINE 


Required> 
wane pewe 
Cn 
(s: Remove] | 


EFishersLSD — 
C Tukey's HSD O Scheffe 
C Sidak C Hochberg's GT2 
Conte [0%5 | 


Selected elfect(s): 


JEJEJ 


You can customize the hypothesis to be tested. Contrasts can be defined across the 
categories of a grouping factor: 


Hypothesis. Select the type of hypothesis. The following choices are available: 
m Pairwise. Compare pairs of groups to determine which pairs differ. 
m FandR Matrices. Tests the hypotheses corresponding to the F and R Matrices tab. 


Adjustment method. The following options are available to compute p-value adjusted 
for multiple comparisons: 
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Bonferroni. Uses Student's ? statistics. It sets the family-wise error rate as 
(1-Confidence) / (Total number of comparisons). 


Fisher's LSD. Equivalent to multiple t tests between all pairs of groups. The 
disadvantage of this test is that no attempt is made to adjust the observed 
significance level for multiple comparisons. 

Tukey's HSD. Uses the Studentized range statistic to make all pairwise 
comparisons. This is the default. 


Scheffé. The significance level of Scheffé's test is designed to allow all possible 
linear combinations of group means to be tested, not just the pairwise comparisons 
available in this feature. Scheffé's test is more conservative than the other tests. 


Sidak. Uses Student's / statistic for pairwise multiple comparisons. 
Hochberg's GT2. Uses Studentized maximum modulus distribution. 


Confidence. Specify the confidence coefficient. The default is 0.95. 
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F and R Matrices 


F and R are the matrices of linear weights contrasting the coefficient estimates for 
fixed and random effects respectively. You can write your hypothesis in terms of the F 
and R matrices. 


Analyze: Mixed Models: Hypothesis Test 


| Main | F RMatices | D Matrix! 


Fixed effect(s} 
| 


penes ecu 


m Fixed effect(s). Specify as many numbers as the dimension of your beta vector. In 
case you specify less, SYSTAT takes the unspecified ones as zero; if you specify 
more, SYSTAT ignores the extra ones. 


m Random effect(s). Specify as many numbers as dimension of your gamma vector. 
In case you specify less, SYSTAT takes the unspecified ones as zero; if you specify 
more, SYSTAT ignores the extra ones. 
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D Matrix 
D is a null hypothesis vector (by default null vector). The D vector, if you use it, must 


have the same number of rows as the F or R matrices. To specify a different D Matrix, 
click the D Matrix tab in the Analyze: Mixed Model: Hypothesis Test dialog box. 


Analyze: Mixed Models: Hypothesis Test 


I id, 
| Main || FA Matrices D Matrix | 


| [Use matrix: 


Example 


010-4 


Specify a vector of dimension same as the number of rows in F and R matrices. 


Estimate. Check this option for testing significance of each contrast (row) of F and R 
matrices individually. This test reports estimate of the estimable linear parametric 
function, its standard error and the corresponding t-test. 
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Using Commands 


First, specify your data with USE filename. Continue with: 


vc 
RESET 
MODEL depvar = INTERCEPT + varlist 
RANDOM varlist 
CATEGORY varlist 
SAVE filename / MRESIDUALS or CRESIDUALS 
or FIXED or VARCOMP or RANDOM or 
SERRORF or DATA or MODEL 
ESTIMATE / METHOD = TYPE1 or TYPE2 or TYPE3 or MIVQUEO or ML or REML 
TYPE - HESSIAN or LIKELIHOOD or PARAMETERS 
CRITERION- RELATIVE or ABSOLUTE 
NEM = nl NNR = n2 CONVERGENCE = dl HALF = n3 
TOLERANCE = d2 CONFI = n4 
GSTART-11,12..1Nv.] 


To perform hypothesis tests: 


HYPOTHESIS 

PAIRWISE varlist / BONF LSD TUKEY SCHEFFE SIDAK GT2 
FMATRIX [matrix] 

RMATRIX [matrix] 

DMATRIX [matrix] 

TEST / CONFI - n ESTIMATE 


Usage Considerations 


Types of data. VC requires a rectangular data file. 


Print options. VC produces extended output if you set the output length to LONG. For 


model estimation, extended output adds fixed effects BLUEs and random effects 
BLUPs. 


Quick Graph. VC produces a Quick Graph of marginal residuals versus marginal 
predicted values. 


Saving files. Several sets of output can be saved to a file. The actual contents of the 


saved file depend on the analysis. Files may include estimated regression coefficients, 
model variables, residuals, and predicted values. 


BY groups. VC analyzes data by groups. 


Case frequencies. VC uses the FREQUENCY variable, if present, to duplicate cases. 
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Case weights. VC uses the values of any WEIGHT variables to weight each case. 


Examples 


Example 1 
Getting Acquainted with the Output Layout 


Here we consider an inheritance study on beef animals of several sire groups (males) 
each mated to a separate group of dams (females). Birth weights of male progeny 
calves were recorded. The datafile KUEHL (Kuehl, 2000), consists of birth weights of 
eight male calves in each of five sire groups. The first column lists the indices of the 
sire groups. In each group there are 8 sires. The second column gives the birth weights 
(in Ibs.). We want to model this variable in terms of the sire effect. 

The five sire groups are randomly chosen from a large population of sire groups. So, 
it is desirable to consider the sire group effect as a random effect. We shall use the 


statistical model 
yg = uta; tE; 


for i-1,...5, and ¡=1,...,8. Here yj is the birth weight (in Ibs.) of the j-th calf produced 
by the i-th sire group, a, being the random effect due to i-th sire group, €, being the 
error term. As is the practice with any mixed effects model, the random effect is 


assumed independent of the random error. It is to be noted that there are two variance 
components in this model, one for sire and one for error term. The input for analyzing 


this one-way random effect model is: 


USE KUEHL 
ve 
CATEGORY SIRE 
MODEL BIRTHW = INTERCEPT 
RANDOM SIRE 
ESTIMATE 
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The output is: 
Categorical values encountered during processing are 
Variables i Levels 
enis pn ra A SN EU C Ne en a AA ES 
SIRE (5 levels) | 177.000 200.000 201.000 202.000 203.000 
Dependent Variable : BIRTHW 
Fixed Covariate(s) : Intercept 
Random Factor (s) : SIRE 
Estimation Method : Residual or Restricted Maximum Likelihood (REML) 
Dimensions 
Covariance Parameters : 2 
Columns in X Ds. 
Columns in Z :7$ 
No. of Observations : 40 


There are two variance components. So the number of variance parameters is 2. X is 
the design matrix for the fixed part. Here it consists of only the intercept term. So it 
consists of just a single column. The five sire groups account for the five columns of 
the random effects design matrix Z. 


Iterations History 


Iteration no. | Iteration type -2L-L Convergence 
SPER A A o e lin 
01 359.579 
1 | ECME 358.651 0.003 
2 | ECME 358.372 0.001 
3 | ECME 358.277 0.000 
4 | ECME 358.242 0.000 
5 | ECME 358.228 0.000 
6 | NR 358.217 0.000 
7 | NR 358.217 0.000 
8 | NR 358.217 0.000 


This table contains information of a somewhat sophisticated nature. The estimation 
method used is REML, which is an iterative method. This table reports the 
convergence status ofthe algorithm. You would rarely need to look at it except to check 
if the estimation has converged at all. For a convergent process the final entry in the 
last column would be zero, as here. Here the convergence has been attained after 12 
steps. The ECME steps can be thought of as part of initialization, which is followed by 
the main Newton-Raphson (NR) steps. The convergence criterion is minus twice log- 
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likelihood, the values of which are reported in the third column. It is possible to change 
this default criterion, by using options for the ESTIMATE command in SYSTAT. 


Fit Statistics 


Final L-L : -179.108 
-2L-L : 358.217 
AIC 362.217 
AIC (Corrected) 362.550 
BIC 365.544 


As discussed earlier smaller values indicate a more parsimonious fit. 


Estimates of Covariance Components 


Variance 
Para 
Variance 
| Parameter 


Error variance 


Estimates of Fixed Effects 


Effect 


Estimate 


463.793 


p-value 


+ 
Intercept | 


Confidence Intervals of Fixed Effects Estimates 


i 95.00% 
| Estimate 


+ 
Intercept | 


Predictions of Random Effects 


H 0.718 
200 i 9.990 
201 i -12.980 
202 H -3.374 
203 1 5.646 


Confidence Intervals of 


Effect Level | Estimate 


Confidence Interval 
Upper 


Random Effects Predictors 


95.00$ Confidence Interval 
r 


Effect Effect Level | Estimate Upper 
o e uam on a RECEN AA 
SIRE 177 i 0.718 4. 15.683 
200 H 9.990 -4.976 24.955 
201 1 -12.980 -27.945 1.985 
202 H -3.374 -18.339 11.591 
203 i 5.646 -9.319 20.611 


These are the estimates and predictions of the fixed and random effects. Note that 
SYSTAT provides point estimates (or predictions) as well as confidence intervals. A 


two-tailed t-test is also reported 


for each effect. The degrees of freedom (df) reported 


for each effect is the error degrees of freedom from the ANOVA table. The confidence 


interval gives an idea about the precision of the estimate. 


The 95% confidence interval 


11-314 
Chapter 6 


for the intercept term, for example, is (66.137, 98.963). This roughly means that the 
actual intercept value (which is unknown) lies within this range with 95% chance. 
Similar explanation applies to the random effect confidence intervals. 


Plot of Residuals vs Predicted Values 


RESIDUAL 


0 100 
ESTIMATE 


SYSTAT always produces this Quick Graph by default. The vertical axis shows the 
residuals and the horizontal axis shows the estimate. The estimate is obtained by 
considering only the fixed effect(s). In this example the only fixed effect is the 
intercept term. So, all the estimates have the same value. This is seen easily from the 
plot, all the points are along the same vertical line. 

The p-value of the intercept is 0.000. So the intercept term is significant at, say, 0.05 
level. This is of course no great news, since all that it says is that the calves weigh 
significantly different from 0 at birth. A more important piece of news lies in the p- 
values for the random effects. All of them are pretty large (more that 0.05, say) and so 
none of the sire effects appear significant. Thus we conclude that the birth weight of a 
calf does not really depend significantly on the group of its dad. 
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Example 2 
A Model with Interaction 


This example illustrates how the VC command handles interaction effects. An 
experiment, described in Milliken and Johnson (1992) was conducted by a company to 
compare performances of three different brands of machines when operated by the 
company's own personnel. Six employees were selected at random and each of them 
had to operate each machine three different times. The file MACHINEI contains these 
data. The data set consists of overall scores that take into account both the quality and 
quantity of output 

We have a two-way treatment structure here. However, since certain operators may 
find certain brands of machines more (or less) difficult to use, we cannot a priori ignore 
the possibility of an interaction between the operator and machine effects. So our 
model is 


yjk = M tat B; + Vij + £i 

where y; is the score of the j-th operator operating the i-th machine at the k-th time 
point. The operator effect D, is assumed random, since the operators were selected at 
random from among the employees. The machine effects a, are fixed. 


The input is: 


USE MACHINE1 

ve 
CATEGORY MACHINE OPERATOR 
MODEL SCORE = INTERCEPT + MACHINE 
RANDOM OPERATOR + MACHINE * OPERATOR 
ESTIMATE 


The following are selections from the output: 


Categorical values encountered during processing are 


Variables Levels A e 


MACHINE (3 levels) + 2.000 3.000 
OPERATOR (6 levels) | xu 2.000 3.000 


4.000 5.000 6.000 


Dependent Variable : SCORE 
Fixed Factor(s) : MACHINE 
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Fixed Covariate(s) : Intercept 
Random Factor (s) OPERATOR, MACHINE*OPERATOR 
Estimation Method : Residual or Restricted Maximum Likelihood (REML) 
Dimensions 
Covariance Parameters : 3 
Columns in X 2.4 
Columns in Z : 24 
No. of Observations : 54 


The 4 columns of X are from the intercept and the 3 machines. The 6 operators and 6 
times 3 interactions account for the 24 columns of Z. 
Estimates of Covariance Components 


Random Effect 


Description Estimate 


OPERATOR | Variance 22.858 
Parameter 
MACHINE*OPERATOR | Variance 13.909 


Parameter 
Variance 0.925 
| Parameter 


Estimates of Fixed Effects 


Effect 


MACHINE 


Confidence Intervals of Fixed Effects Estimates 


95.00% Confidence Interval 


Effect Level ! Estimate Lower Upper 
Sc A A itia a E tte 
Intercept H 66.272 60.733 71.811 
A NUM  —À—— Beatus uni oe rs tein 
MACHINE 1 p 233.917 718.767 -9.066 

2 H -5.950 710.801 71.099 

3 i 0.000 . . 


Notice that the estimate for the last machine is 0. Actually this is a statistical artifact to 
avoid non-estimability problems. It is not possible to estimate all the three a, 's 
together. We need to put some extra condition. SYSTAT imposes the condition that the 
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last œ; equals 0. The same assumption is reflected in the confidence intervals also that 
are reported next. 


Predictions of Random Effects 


Effect Level | 


MACHINE*OPERATOR peat 
X 


3*5 
3*6 


Confidence Intervals of Random Effects Predictors 


1 95.00% Confidence Interval 
Estimate Lower Upper 


H 
1 
i 
i 
1 
i 
2*3 1 
1 
i 


2.487 2.388 36 1.042 0.305 


Effect Effect Level 


MACHINE*OPERATOR 


3*6 


A quick glance through the p-value column tells us which of the random effects are 
significant. Smaller p-values are more significant. If we are using level 0.05, then we 
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should look out for p-values smaller than 0.05. However, there is a caveat. When 
looking for significant effects we must always start from the higher order effects first. 
In this example, these are the interactions. Looking for significant lower order effects 
make sense only when higher order effects are all insignificant, In our example, 
however, there is significant interaction between machine2 and operatoró. Judging 
from the large negative value of the t-statistic, operator6 was having some real 
difficulty with machine2. 


Type III Tests for Fixed Effects 


Effect | Numerator df Denominator df F-ratio 


MACHINE | 2 10 20.576 0. 


This ANOVA table is for the fixed effects (i.e., machines in our example). It tries to tell 
us if the machines are significantly different. But wait! We have already found the 
presence of significant interaction. So we must not jump to the conclusion that the 
machines differ significantly just by looking at the low p-value. The apparent 
difference between the machines might very well be caused by operator6 messing up 
with machine2. (Remember the significant interaction term for this pair?) A good 
analyst should first investigate more carefully the significant interaction terms before 
blindly testing main effects. 

Notice that this model does not take time into account. However, it may happen that 
à machine behaves differently when run for the first time than when it is run next. If 
we have reason to suspect this, then we should introduce an interaction term between 
machine and time. 


The input is: 


USE MACHINE1 
vc 
CATEGORY MACHINE OPERATOR 
MODEL SCORE - INTERCEPT « MACHINE 
RANDOM OPERATOR + MACHINE * OPERATOR + MACHINE * TIME 
ESTIMATE 


We have made the new interaction term a random effect assuming that the times were 
chosen at random. This time we shall not present the entire output (which is rather 
long). Instead, we shall show only the relevant portion, viz., the interaction effects with 
time. 
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The following are selections from the output: 


Predictions of Random Effects 


Effect Effect Level 


MACHINE* OPERATOR 1*1 
1 


MACHINE* TIME 1*l 


3*2 
3*3 


| Estimate Standard Error 


0. 
i 0. 
1 0. 
i 0.000 . . ` 
1 0.000 0.008 30  -0.013 0.990 
H 0.000 0.008 30 0.011 0.991 
4 0.000 0.008 30 0.003 0.998 
1 0.000 0.008 30 0.012 0.990 
y 0.000 0.008 30  -0.015 0.988 


Confidence Intervals of Random Effects Predictors 


Effect 


MACHINE*OPERATOR 


i 
Effect Level | Estimate 
+ 


4 95.00% Confidence Interval 
Lot Upper 
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MACHINE*TIME 1*1 


:000 -0.016 0.016 


3*3 i 


Our interest lies in whether the interaction of machine with time is significant or not. 
Well, judging by the high p-values they are not. So we can indeed remain happy with 
the first model. For a more sophisticated analysis of the same data set please see the 
next chapter. 

You might be tempted to put all possible interaction terms in your model to 
safeguard against overlooking any potential interactions. However, remember that 
interaction terms introduce more parameters, which eat up degrees of freedom. So the 
more interaction terms you introduce, the less degrees of freedom are left for 
estimating the variance components, resulting in less precise estimates. 


Example 3 
Nested Effects 


This is an example where one effect is nested inside another, i.e., the levels of one 
effect have different meanings within the levels of another effect. 

In this data set, from Kuehl (2000), our interest lies in comparing two standard 
pesticide methods. In particular, we want to find out if the amount of residue left on 
cotton plant leaves is the same for the two methods, which we shall call methods | and 
2. To test this 6 batches of plants were sampled from the field, Two plants were used 
in the experiment from each batch. Thus, there were 12 plants in the experiment. The 
plants inside each batch were from the same field plot. Method 1 was applied to 3 
randomly selected batches, and the remaining 3 batches were given Method 2. The 
amount of residue on leaves was measured after a specified amount of time for each of 
the 12 plants. Data are in PESTRESIDUE file. 


We shall fit the model 


Jjk = M tat Bici + Eijk 
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where ¡=1, 2, j=1, 2, 3 and k=1, 2. Here y;;y is the measurement for the k-th plant in the 
j-th batch under i-th method. 


The input is 


USE PESTRESIDUE 

ve 
CATEGORY METHOD BATCH 
MODEL Y = INTERCEPT + METHOD 
RANDOM BATCH (METHOD) 
ESTIMATE 


The following are selections from the output: 


Categorical values encountered during processing are 


Variables 
METHOD (2 levels) 1.000 .000 
BATCH (6 levels) SH 2.000 3.000 4.000 5.000 6.000 


Dependent Variable : Y 

Fixed Factor(s) : METHOD 

Fixed Covariate(s) : Intercept 

Random Factor (s) : BATCH (METHOD) 

Estimation Method : Residual or Restricted Maximum Likelihood (REML) 


Dimensions 


Covariance Parameters : 4 
Columns in X 3 
Columns in Z x. 6 
No. of Observations 112 


The three columns in X are due to the intercept and the two methods. There are 3 times 
2 batch (method) nested terms giving rise to the 6 columns of Z. 


Fit Statistics 


Final L-L : -38.503 
-2L-L 77.005 
AIC : 81.005 
AIC(Corrected) : 82.720 
BIC : 81.610 


Estimates of Covariance Components 
Estimate 


Variance 67.500 


Variance 55.083 
Parameter 


+ 
| Parameter 
+ 
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Notice that random variation present in the data originates more from the difference of 
the batches within the methods than random errors (e.g., measurement errors, 
differences among the plants within the batches etc). 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t p-value 

o ona sd IPTE cc a tr ie a 

Intercept i 69.833 5.629 4 12.407 0.000 

Sees Bone tolam ccs -. 0 e CO o o os prat 

METHOD 1 i 50.167 7.960 4 6.302 0.003 
2 H 0.000 0.000 . 


Confidence Intervals of Fixed Effects Estimates 


i 95.00% Confidence Interval 
Estimate Lower Upper 


Effect Level | 

suci itane: tion add ien SER Tcr e S oer ccelis ccc 

Intercept H 69.833 54.206 85.461 

a aes pala RD A Do A aa 

METHOD 1 i 50.167 28.066 72.267 
2 i 0.000 . 


The small p-values (less than 0.05, say) indicate significant terms. The dots in the last 
row are because of the estimability condition imposed on the fixed effect coefficients 
by SYSTAT: the last coefficient of each fixed main effect is assumed to be 0. The small 
p-value for the first method tells us that the two methods indeed differ significantly. 
However, before believing the p-values we must check the higher order terms (the 
nested terms, in this example). This is done next. 


Predictions of Random Effects 


Standard Error p-value 


BATCH (METHOD) 1) 


6 6 
20) 5.962 6  -1.191 
3(1) 5.962 6 1.787 
4(2) 5.962 6 0.139 
5(2) 5.962 6 0.377 
6(2) 5.962 6 -0.516 


Confidence Intervals of Random Effects Predictors 


95.00% Confidence Interval 


Effect Effect Level | Estimate Lower Upper 
Up atera ARE Re minnie utu 4 A a PEN eh 
BATCH (METHOD) 1(1) H -18.139 11.036 
20) f -21.690 485 
30) { -3.934 25.241 
4(2) ! -13.759 15.416 
5(2) -12.338 836 
6(2) -17.665 510 


None of the random coefficients are significant, since the p-values are quite large 
(larger than 0.05, say). So it makes sense to look into the main effects. These were 
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tested in fixed effect table given earlier. It showed that the methods indeed differed 
significantly. 

Type III Tests for Fixed Effects 

Effect | Numerator df Denominator df F-ratio p-value 


METHOD ; 1 4 39.720 0.003 


Once more we see that the methods produce significantly different measurements from 
the Type III Tests for the Fixed Effects table. 


Example 4 
Split Plot Design 


The data set for this example comes from Milliken and Johnson (1992). It is an 
agricultural data set obtained from a split plot design laid out as follows. We want to 
compare four fertilizers and two varieties of crops. We have 4 (whole) plots to try these 
on. These are grouped into two blocks. The two varieties are assigned randomly to the 
two (whole) plots in each group. Each whole plot is split into 4 subplots, and the4 
fertilizers are applied randomly to these. 


The yield of crop for each subplot is noted. The data are given in CROPS data file. 
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The input is: 


USE CROPS 
ve 
CATEGORY BLOCK VARIETY FERT 
MODEL YIELD = INTERCEPT + VARIETY + FERT + VARIETY * FERT 
RANDOM BLOCK VARIETY * BLOCK 
ESTIMATE 


The following are selections from the output: 


Fit Statistics 


Final L-L 2 719.310 
-2L-L : 38.619 
AIC : 44.619 
AlC(Corrected) : 50.619 
BIC 2 44.857 


Estimates of Covariance Components 


Random Effect Description Estimate 


t 
p———— adi 
BLOCK | Variance 16.087 
| Parameter 
A di 
VARIETY*BLOCK ¡ Variance 0.061 
| Parameter 
PP ete A 
Error variance | Variance 2.159 


Notice how small the interaction variance component is compared with the other two. 
This signals that we could as well ignore the interaction. But we should wait for the 
latter parts of the output for more clinching evidence. 


Estimates of Fixed Effects 


+ 
+ 


Estimate Standard Error df t p-value 


VARIETY*FERT 
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Confidence Intervals of Fixed Effects Estimates 


i 95.00% Confidence Interval 
Estimate Lower Upper 


Effect Level 


VARIETY 


VARIETY*FERT 


First let us understand why there are so many dots in this table. These are because of 
the estimability condition enforced by SYSTAT. The last coefficient of each fixed main 
effect is assumed to be 0. Also, many of the interaction terms have to be assumed 0 to 
keep the others estimable. The p-values are not reported for these forced zeros, since 
they would not make any sense there. The reported p-values for the interaction terms 
are all insignificant (larger than 0.05, say). As we shall presently see from a latter table, 
the random interactions are also insignificant. Let us now look at the main effects. The 
fertilizer effects are all significant (at 0.05 level) but the variety effects are not 


significant. 
Predictions of Random Effects 
Effect Effect Level | Estimate df t p-value 


Type III Tests for Fixed Effects 
| Numerator df Denominator df  F-ratio 


Effect 1 Numerator MP Deno A e em 
VARIET "ler 1 1 1 0.937 
TU x l 3 6 6.205 
VARIETY*FERT | 3 6 0.239 


The ANOVA table usually provides a summary of what we have already found from 
the other tables. Here we see that the fertilizers differ significantly, while the other 


effects are insignificant. 
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Example 5 
Using Covariates 


This example is based on a clinical data set presented in Hocking (2003), where a 
pharmaceutical firm wants to test a new drug for a particular disease. The response is 
a measure of the improvement in the patient's status. A sample of 3 clinics is selected 
at random from a large population of clinics. From each clinic a sample of 10 patients 
with the particular disease are selected. The drug is applied to each patient and a 
response (Y) of the drug and a relevant physical characteristic (Z) for each patient, are 
recorded. The CLINCOV file contains this data set. 


We want to fit the following model to this data set. 


Yy 5 uta; t+ Bzyt ey 


where cz, 's are the only random effects. The aim is to see if the drug is really effective 
or not, and whether the clinics influence the effect of the drug significantly. We want 
to guard against accidentally attributing any change in a patient's status to the drug if 
the change is actually due to the relevant physical characteristic of the patient. That is 
why we have included Z in our model. 


The input is: 


USE CLINCOV 
vc 
CATEGORY CLINIC 
MODEL Y = INTERCEPT + Z 
RANDOM CLINIC 
ESTIMATE 


The following are selections from the output: 


Dimensions 
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The first column in X is due to the intercept term, while the second comes from the 
covariate. 


Fit Statistics 


Final L-L i -75.717 
-2L-L : 151.434 
AIC : 155.434 
AIC (Corrected) : 155.914 
BIC : 158.099 


Estimates of Covariance Components 


Estimate 


Random Effect | 


Error variance | Variance 
| Parameter 


The first variance component is estimated to be 0 up to three decimal places. Compared 
to the much larger error variance this already foreshadows the insignificance of the 
clinic effect. 


Estimates of Fixed Effects 
Standard Error 


Estimate 


z i . 0.083 


Confidence Intervals of Fixed Effects Estimates 


95.00% Confidence Interval 
Lower Upper 


Both the intercept and the slope terms are highly significant as judged by the small p- 
values. The significance of the intercept says that the drug has nontrivial effect, while 
the significant slope indicates that the patients' responses depend significantly on their 


physical characteristics as well. 


Predictions of Random Effects 


Effect Effect Level | 


0.000 


CLINIC 1 Y A 5 

2 i 0.000 0.028 26 0.009 0.993 

3 H 0.000 0.028 26  -0.010 0.992 
Predictors 


Confidence Intervals of Random Effects 
i 95.00% Confidence Interval 


Effect Effect Level i Lower Upper 
CLINIC 1 0.057 
0.057 
0.057 
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The clinic effects are all insignificant. So the clinics do not differ much among 
themselves in the present context. 

Type III Tests for Fixed Effects 

Effect | Numerator df Denominator df F-ratio p-value 


ss eo con NA nieces sioner es P! oam, 


2 i 1 26 46.277 0.000 


So far we have accepted the covariate as something informative. But is the covariate 
really bringing relevant information? The small p-value in the above F-test assures us 
this in the affirmative. Had this test shown the effect of Z to be insignificant we would 
have done better by dropping it from our model. 


Example 6 
Unbalanced Data: Different Types of ANOVA 


This example is meant to show the difference among the three ANOVA methods: 
TYPEI, TYPE2 and TYPE3. These three methods will always give the same results if 
the data set is balanced, i.e, if there are equal number of observations in each cell. (For 
a nested design, a balanced data set also requires the same number of nested levels 
under each nesting effect.) So to see the difference among the three methods we need 
an unbalanced data set. 

The MACHINE? data, from Milliken and Johnson (1992) p.285 presents an 
unbalanced data set where two machines are being operated by six randomly selected 
operators. Each operator is allowed to operate each machine at most three times. 

Here machine is a fixed effect and operator is a random effect. In this unbalanced 
case the three methods can lead to different outcomes. 


First we shall apply the TYPE! method. 


The input is: 


USE MACHINE2 

ve 
CATEGORY MACHINE OPERATOR 

MODEL SCORE = INTERCEPT + MACHINE 
RANDOM OPERATOR + MACHINE*OPERATOR 
ESTIMATE/ METHOD = TYPE1 


Refer to chapter on “Linear Models“ on page 1, Statistics II fora description of various 
types of sum of squares, We shall focus our attention on only the part of the output that 
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highlights the difference between the three types of ANOVA estimation. The rest of the 
output is not shown. 


The following are selections from the output: 


The categorical values encountered during processing are 


Variables Levels 

MACHINE (2 levels) 1.000 2.000 

OPERATOR (6 levels) | 1.000 2.000 3.000 4.000 5.000 
; 6.000 

Dependent Variable : SCORE 

Fixed Factor(s) : MACHINE 

Fixed Covariate(s) : Intercept 


Random Factor (s) 


OPERATOR, MACHINE*OPERATOR 


Estimation Method : ANOVA Type I 
Dimensions 

Covariance Parameters : 3 
Columns in X :.3 
Columns in Z + 18 

No. of Observations : 26 


Type I Sum of Squares 


Source 


Q(MACHINE) + 


MACHINE 
0.136*V(OPERAT- 
OR) * 
Se 4.219*V (OPERAT 
OR) * 
2.176*V (MACHIN- 
i 
== Mu ee 43*V (MACHIN- 
MACHINE*OPERATOR | E*OPERATOR) * 


A TE plas 
erm involving parameters of the fixed effects as indicate 


* Q(): Quadratic ti 


This is a table that is produced whenever an ANOVA estimation method (of any type) 
is used. The first four columns are just as in any ANOVA table. The last column tells 
us what each MS is estimating unbiasedly. This information is important for making 

sense out of the ANOVA F-tests that are presented later. For unbalanced data the MS 
for each effect may have some contribution from some other effects mixed with it. This 
depends on the nature of imbalance and the type of ANOVA used. Here, for example, 
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the MS for machine has part of the operator effect and machine-operator interaction in 
its expectation. 


Error Terms 
Denominator 


Effect Expression Error Term 


0.032 71.875 
MS(OPERATOR) + 
1.162 
MS (MACHINE*OPE- 
R.. 


1.065 61.302 
MS (MACHINE*OPE- 

RATOR) - 0.065 

MS (Error) 


+ 
e la e 
* 


MACHINE*OPERATOR MS (Error) 0.880 


Analysis of Variance 


Numerator df Denominator df Mean Squares F-ratio 


i 359.434 1 5.734 359.434 5.001 
OPERATOR i 797.665 5 4.991 159.533 2.602 
MACHINE*OPERATOR | 288.030 5 14.000 57.606 65.497 
ERROR i 12.313 14 0.880 


Analysis of Variance (contd...) 


Source | p-value 
DEL T aT EO en 
MACHINE H 0.069 
OPERATOR 4 0,159 
MACHINE*OPERATOR | 0.000 
ERROR 1 


Here is the ANOVA table. Only the interaction effect appears significant (p-value less 
than 0.05, say). Since we are using TYPE] ANOVA, this means that interaction is 
significant after taking out the main effects of the machines and operators. The 
insignificant machine effect means that the machine effect is insignificant after taking 
out the operator effect and interaction effect. Thus, in a sense, the TYPE! method is 
already taking the interaction into consideration before reporting the main effects. 
Type I Tests for Fixed Effects 

Source | Numerator df 


-T-------- 4---------- 
MACHINE | 


F-ratío p-value 


The two tables above may appear contradictory at first. Both seem to test the machine 
effect, yet produce different p-values! The difference is explained by the fact that the 
first table performs the test with the random effects in mind. That is why the 
denominator degrees of freedom is fractional. However, the second table carries out 
simple fixed effect ANOVA. 
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Next let us use the TYPE2 method. 


The input is: 


USE MACHINE2 

ve 
CATEGORY MACHINE OPERATOR 
MODEL SCORE = INTERCEPT + MACHINE 
RANDOM OPERATOR + MACHINE*OPERATOR 
ESTIMATE/ METHOD = TYPE2 


The following are selections from the output: 


Dependent Variable : SCORE 


Fixed Factor (s) : MACHINE 
Fixed Covariate(s) : Intercept 
Random Factor (s) : OPERATOR, MACHINE*OPERATOR 


Estimation Method : ANOVA Type II 


Type II Sum of Squares 
df EI Mean Squares 


Source 


1 284.311 284.311  Q(MACHINE) + 
2.318*V(MACHIN- 
E*OPERATOR) * 


+ 

1 

! 

i 

+ 

beast 791.665 159.533 — 4.219*V(OPERAT- 
1 OR) * 

i 2.176*V (MACHIN- 
+ 
+ 
c 


5 288.030 57.606 43*V (MACHIN- 
E*OPERATOR) * 
V(Err 


This table shows the expected values of the ANOVA MS. A quick comparison with the 
corresponding table for TYPE 1 would show that the interaction MS and the operator 


MS are the same, but the machine MS is now different. 


Error Terms 


Denominator 
Effect Express 


1.135 
MS (MACHINE*OPE- 
RATOR) - 0.135 
MS (Error) 
1.065 
MS (MACHINE*OPE- 
RATOR) - 0.065 
MS (Error) 


MS (Error) 0.880 


MACHINE*OPERATOR 
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Analysis of Variance 


Source | Type II SS Numerator df 
A TEE ered ad a SS cuc aptota qct DE 
MACHINE 1 284.311 1 
OPERATOR i 797.665 5 
MACHINE*OPERATOR | 288.030 5 
ERROR i 12.313 


Source -value 
MACHINE 0.091 
OPERATOR 0.159 
MACHINE*OPERATOR | 0.000 
ERROR 


Type II Tests for Fixed Effects 


Source | Numerator df Denominator df F-ratio p-value 


A EE A 


MACHINE | 1 4.982 6.369 0.053 


Finally, here is what happens with the TYPE3 method. 


The input is: 


USE MACHINE2 

ve 
CATEGORY MACHINE OPERATOR 
MODEL SCORE = INTERCEPT + MACHINE 
RANDOM OPERATOR + MACHINE*OPERATOR 
ESTIMATE/ METHOD = TYPE3 


The following are selections from the output: 


Dependent Variable : SCORE 


Fixed Factor (s) : MACHINE 
Fixed Covariate(s) : Intercept 
Random Factor (s) : OPERATOR, MACHINE*OPERATOR 


Estimation Method + ANOVA Type III 


Mean Squares 
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Dimensions 


Covariance Parameters : 3 
Columns in X : 
Columns in Z : 18 


No. of Observations : 26 


Type III Sum of Squares 


Source H df ss Mean Squares Expected MS 
aE Ie dem A A Mm MEE 


i 1 324.803 324.803 Q(MACHINE) + 
i 1.800*V(MACHIN- 
i E*OPERATOR) + 


MACHINE 


OPERATOR 5 778.751 155.750 4.086*V(OPERAT- 
OR) * 
2.043*V (MACHIN- 


57.606 —2.043*V(MACHIN- 
E*OPERATOR) * 
V(Error) 


* Q(): Quadratic term involving parameters of the fixed effects as indicated 


Here the MS for both the main effects are different from those for types 1 and 2. 
However, the interaction MS is the same for all the three types. This is because, for all 
the three types the interaction SS is computed after taking out the contribution of the 


other two effects. 


Error Terms 


Denominator 
Expression Error Term 


0.881 50.859 
MS (MACHINE*OPE- 
RATOR) * 0.119 
MS (Error) 


Effect 


4 
i 

+ 
1 


1.000 
MS (MACHINE*OPE- 
RATOR) * 0.000 


1 


MACHINE*OPERATOR | MS (Error) 0.880 


Analysis of Variance 


Source 

MACHINE n 1 324.803 6.386 
OPERATOR i 155.750 2.704 
MACHINE*OPERATOR | 288.030 57.606 65.497 
ERROR ; 12.313 0.880 


Analysis of Variance (contd...) 


Source 
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MACHINE 3 0.053 
OPERATOR i 0.150 
MACHIME*OPERATOR | — 0.000 
ERROR 1 
Type III Tests for Fixed Effects 
Source | Numerator df Denominator df F-ratio  p-value 
acm | 1 5.021 6.369 0.053 
Example 7 
» 
Exploring with Residuals 


After drying beech wood the humidity level at any given point inside a plank typically 
depends on the depth of the point. In this example we want to study the relation 
between the humidity level (measured as a percentage) with the depth for 20 different 
randomly selected beech planks. For each plank we measure the humidity level for 5 
depths and 3 widths, The PLANKS data file contains this data set. 


We want to model the data as follows: 


Yik = B+ at Bry +h +e, 


where i=1,2,3, j=1,...,5 and k-1,...,20. Here the a's denote the depth effect, the B's 
denote the width effects and 5's denote the plank effects. We have also allowed 
interaction between the depth and width effects. The interaction effect is denoted by 
the y's . We do not want our inference to involve the particular sample of 20 planks 
used in the experiment. So we consider the plank effect as random. 


CATEGORY DEPTH WIDTH PLANK 
MODEL HUMIDITY = INTERCEPT + WIDTH + DEPTH + WIDTH*DEPTH 
RANDOM PLANK 


ESTIMATE 


The following are selections from the output: 
Dimensions 


Covariance Parameters : 3 
Columns in X : 24 
Columns in Z : 20 
No. of Observations : 300 
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The 24 columns in X have the following genesis: 1 column from the intercept term, 3 


from the width coefficients, 5 from depth, and 3 times 5 from the interactions. 


Fit Statistics 


Final L-L : -332.029 
-2L-L : 664.058 
AIC : 668.058 
AlC(Corrected) : 668.100 
BIC : 675.363 


Estimates of Covariance Components 
Est. 


Random Effect 


PLANK i Variance 
| Parameter 

Error variance | Variance 0.404 
Parameter 


Estimates of Fixed Effects 
Effect Level | Estimate Standard Error df t p-value 


cee een een enna EM AME E 


4.395 0.263 19 16.710 0.000 


panne d em tem m tin i mam sim nit iia ee 


0.137 


WIDTH*DEPTH TTE 
1 


387 
se 
Confidence Intervals of Fixed Effects Estimates 


95.00% Confidence Interval 
Estimate Lower Upper 


~ 
" 
3 

M ee fe ee ee O E 
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ro €—— A e 
WIDTH*DEPTH 1*1 i -0.230 -0.789 0.329 
193 :4 0.220 -0.339 0.779 
1*5 i 0.325 -0.234 0.884 
in? i 0.260 -0.299 0.819 
O. 0 0.000 
2 O 0.025 -0.534 0.584 
2*3. Y 0.520 -0.039 1.079 
2*5 i 0.355 -0.204 0.914 
227 i 0.160 -0.399 0.719 
2*9 i 0.000 
2*5 i 0.000 
23 H 0.000 
3558 j} 0.000 
3*7 i 0.000 . 
3*9 i 0.000 


This table shows, among other things, the results of the t-tests for each fixed effect 
coefficient. As usual we start by considering the higher-order effects first. The 
interactions are all insignificant (p-values above 0.05, say). The dots in some of the 
rows owe their origin to the estimability restriction imposed by SY STAT. Next we look 
at the main effects. Width2 appears to be only significant width term. The fact that the 
p-value for Width] is large means Width! does not differ significantly from Width3, 
which is the reference Width (since SYSTAT assumes that the coefficient for Width3 
to be 0 as an estimability restriction). A similar argument shows that coefficients of 
Depths! and 9 do not differ significantly, while the other depth coefficients do. 


Predictions of Random Effects 


Effect Effect Level | Estimate Standard Error df t p-value 
Scanian sme ae upto lh sae cng mona bn nL ape bad ee mls fl neem os UE UR incite 
PLANK 1 i 0.272 266 -3.245 0.001 
2 i 0.272 266 -2.195 0.029 
3 i 0.272 266 -0.715 0.475 
4 y 0.272 266 0.765 0.445 
5 t 0.272 266 -3.723 0.000 
6 { 0.272 266 1.028 0.305 
y i 0.272 266 -0.333 0.739 
8 1 0.272 266 -3.078 0.002 
9 1 0.272 266 -4.797 0.000 
10 | 0.272 266 6.805 0.000 
11 H 0.272 266 -1.789 0.075 
12 H 0.272 266 0.455 0.650 
13 i 0.272 266 3.319 0.001 
14 i 0.272 266 1.720 0.087 
15 1 0.272 266 6.924 0.000 
16 i 0.272 266 2.627 0.009 
17 f 0.272 266 -4.033 0.000 
18 i 0.272 266 3.845 0.000 
19 i 0.272 266 -5.346 0.000 
20 i 0.272 266 1.768 0.078 


Confidence Intervals of Random Effects Predictors 


95.00% Confidence Interval 
L 


Effect Level Upper 


nani 


-1.012 -1.547 70.477 
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6 i 
7 i 
8 ; 
9 ] 
10 1 
11 1 
12 f 0.124 
13 [ 0.902 
14 y 0.467 
15 l 1.882 
16 ! 0.714 
17 H -1.096 
18 t 1.045 
19 }  =1.453 
20 j 0.480 
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0.814 
0.445 
-0.302 
-0.769 
2.385 
0.049 
0.659 
1.437 
1.003 
2.417 
1.249 
-0.561 
1.580 
-0.918 
1.016 


Most of the plank effect coefficients are significant (p-values below 0.05, say). 


Type III Tests for Fixed Effects 


Effect Denominator df 
WIDTH 266 
DEPTH i 266 
WIDTH*DEPTH | 266 


The interaction effect is not significant. But 
significant. So we drop the interaction from 


The input is: 


ve 
CATEGORY DEPTH WIDTH PLANK 


F-ratio p-value 


29.646 0.000 
78.259 0.000 
1.084 0.375 


the other two main fixed effects are 
the model and refit. 


MODEL HUMIDITY = INTERCEPT + WIDTH + DEPTH 


RANDOM PLANK 
SAVE MYRESIDS / MRESIDUALS 
ESTIMATE 


Notice that here we are 


saving the marginal residuals in a data set called MYRESIDS, 


which will be automatically created by SYSTAT. This data set will have two variables: 


estimate and mresiduals. 


The following are selections from the output: 


Fit Statistics 


Final L-L : -331.912 
-2L-L : 663.824 
AIC : 667.824 
AIC (Corrected) : 667.865 


BIC : 675.184 
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Estimates of Covariance Components 


Random Effect 


| Variance 0.405 
| Parameter 


Error variance 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t  p-value 


Confidence Intervals of Fixed Effects Estimates 


95.00% Confidence Interval 
r 


This table shows, among other things, the results of the t-tests for each fixed effect 
coefficient. Notice that dropping the interaction term has changed things considerably. 
Coefficients for Widths1 and 3 are now significantly different. However, the 
coefficients of Depths! and 9 still do not differ significantly, while the other depth 


coefficients do. 

Predictions of Random Effects 

Effect Effect Level | Estimate Standard Error df t p-value 

PLANK 1 -0.882 274 73.244 0.001 
2 -0.597 274 72.194 0.029 
3 t -0.194 274 -0.715 0.475 
4 0.208 274 0.765 0.445 
5 ; -1.012 274 73.721 0.000 
6 i 0.279 0. 274 1.027 0.305 
7 H =0.091 0. 274 70.333 0.739 
8 i -0.837 0. 274 73.077 0.002 
9 71.304 0. 274 74.795 0.000 
10 1.849 0. 274 6.802 0.000 
11 -0.486 0. 274 71.788 0.075 
12 0.124 0.272 274 0.455 0.650 
13 0.902 0.272 274 3.318 0.001 
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14 i 0.467 0.272 
15 i 1.882 0.272 
16 i 0.714 0.272 
17 i -1.096 0.272 
18 1 1.045 0.272 
19 i -1.453 0.272 
20 i 0.480 0.272 


Confidence Intervals of Random Effects Predictors 


Effect t Level | Estimate Lower 
A m pce ee fiie 
PLANK H -1.417 
i -1.132 
1 -0.730 
i -0.327 
i -1.547 
f -0.256 
H -0.626 
I -1.372 
I 71.304 -1.839 
i 1.849 1.314 
f -0.486 -1.022 
i 0.124 -0.412 
i 0.902 0.367 
i 0.467 -0.068 
i 1.882 1.347 
H 0.714 0.179 
H -1.096 -1.631 
i 1.045 0.510 
H 71.453 -1.988 
H 0.480 -0.055 
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274 1.719 0.087 
274 6.921 0.000 
274 2.626 0.009 
274  -4.031 0.000 
274 3.843 0.000 
274  —-5.344 0.000 
274 1.767 0.078 


95.00% Confidence Interval 


Upper 


Most of the plank effect coefficients are significant (p-values below 0.05, say). 


Type III Tests for Fixed Effects 


Effect | Numerator df Denominator df F-ratio 
WIDTH 2 274 29.574 
DEPTH 4 274 78.068 


Next we shall look at the residual plot, where residuals are plotted against estimates. If 
the Quick Graph feature is turned on then the plot is automatically produced. 
Otherwise you may create the plot by an explicit PLOT command as shown below. 
Incidentally, if you are using the Quick Graph feature then you do not need to save the 


residuals as we did above. 


To make the plot directly, 


The input is: 


USE MYRESIDS 
PLOT MRESIDUAL * ESTIMATE 
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The output is: 
Plot of Residuals vs Predicted Values 
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Example 8 
Missing Data 


This example shows how we can deal with missing values in a data set using SYSTAT. 
The data set we shall use is from Hocking (2003), where a pharmaceutical company is 
tying to test a new medicine. Three clinics have been selected at random from a large 
number of clinics. The drug is administered to 10 randomly selected patients, 
However, some of the measurements from some of the clinics have not been reported. 
The data are setup in the file PATMISS. 

Before we can fit a statistical model to this data set, we need to have an idea about 
why some of the observations are missing. For instance, it may be because of some 
transcription problem which may be assumed to be independent of the Observations. 
This called Missing Completely At Random (MCAR), and is the most frequently made 
assumption in case we are ignorant about the exact cause behind the data loss. 
SYSTAT, like most other software, analyzes this special case only. If, however, the 
missing observations correspond to patients for whom the drug have caused serious 
side-effects leading to cancelling the medication, then the analyst had better investigate 
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the nature of the side effects, rather than continue happily with the MCAR assumption 
about the incomplete data set. 

In this data set, however, no such cause is reported against a reasonable use of the 
MCAR assumption. 


The input is: 


USE PATMISS 
ve 
CATEGORY CLINIC 
MODEL Y = INTERCEPT + Z 
RANDOM CLINIC 
ESTIMATE 


Notice that no mention is made about the data being incomplete. This is because, 
SYSTAT sees the incomplete nature of the data set from the data set itself, and then it 
automatically uses the appropriate analysis using the MCAR assumption. 


The output is: 

Categorical values encountered during processing are 
Variables i Levels 
CLINIC (3 levels) | 1.000 2.000 3.000. 


Dependent Variable : Y 
Fixed Covariate(s) : Intercept 


Random Factor (s) : CLINIC 
Estimation Method : Residual or Restricted Maximum Likelihood (REML) 


Dimensions 
Covariance Parameters : 2 
Columns in X : 1i 
Columns in Z ue 
No. of Observations : 20 
Iterations History 


Iteration no. 


Umi ake pue d + 

01 . 

1 | ECME 123.725 0.003 
2 | ECME 123.537 0.002 
3 | ECME 123.443 0.001 
4 | ECME 123.392 0.000 
5 | ECME 123.362 0.000 
6 | NR 123.314 0.001 
7 | NR 123.311 0.000 
8 | NR 123.311 0.000 
9 | NR 123.311 0.000 
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Fit Statistics 


Final L-L : -61.655 
-2L-L 123.311 
AIC 127.311 
AIC (Corrected) 128.061 
BIC : 129.200 


Estimates of Covariance Components 


Random Effect | Description E: 


CLINIC | Variance 
| Parameter 
————— i earn steep UM 2 
Error variance | Variance 30.222 
| Parameter 


Estimates of Fixed Effects 


Effect | Estimate Standard Error df t p-value 


Intercept | 6.762 1.930 2 3.504 0.073 


Confidence Intervals of Fixed Effects Estimates 


i 95.00% Confidence Interval 
Effect | Estimate Lower 


Intercept | 6.762 -1.541 15.066 


Predictions of Random Effects 


Effect Effect Level 


CLINIC 1 -0.986 1.932 17 -0.510 0.616 
-1.162 1.984 17 -0.585 0.566 
3 2.148 2.047 17 1.049 0.309 


Confidence Intervals of Random Effects Predictors 


H 95.00% Confidence Interval 
Effect Effect Level | Estimate Li 


wer Upper 
CLINIC 1 3.090 
H 3.025 

3 i 2.148 -2.171 6.467 


This example shows that the SYSTAT output for missing data is as simple as that for 
complete data. We see that the clinics do not differ significantly among themselves. 
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Linear Mixed Models 


Arnab Chakraborty, Ravindra Jore, Bindu-Madhav Yeelarthi, and Javed Pathan 


Linear Mixed Models (LMM) fits and analyzes mixed models with structured 

covariance/correlation matrices for random effects and residuals. Variance 

Component, Compound Symmetry, Diagonal, and Unstructured are the four types of 

structures provided for random effects. Variance Components, Compound Symmetry, 

and Auto-Regressive(1) are the three types of structures provided for error 

covariances. Various models like random intercept model, random coefficients 

model, variance components model, mixed effects ANOVA model, and models with 

autocorrelated errors can be fitted using LMM. LMM allows random effects to be 

both categorical and continuous. In LMM, SYSTAT provides two methods to estimate 

covariance parameters, viz., Maximum Likelihood and Restricted (Residual) 

Maximum likelihood. SYSTAT provides: 

m Covariance parameter estimates 

m Fixed effect and random effect predictors, standard errors, confidence intervals 
and t-test for testing whether these estimates/predictors are significant. 

m F-ratio tests for fixed effects. 

" Log-likelihood, Akaike Information Criterion (AIC) , Akaike Information 
Criterion Corrected (AICc) and Bayesian Information Criterion (BIC), and 


iteration history as default output. Save option is provided to save residuals, 
predictions, model parameter estimates and their standard errors to a specified 


data file. 
m Plot of residuals against estimates 
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Statistical Background 


A general linear mixed model is a model of the form 


y= XP+Ziyj+...+Zy,+e 


where y is a response vector, X and Z¡'s are known matrices (either design matrices or 
covariate matrices), [3 is the vector of fixed effects, each y, is a vector of random 
effects, and e is the random error vector, Here y is a random vector, whose randomness 
comes partly from the random vector y and partly from  . We assume that the random 
vectors y, and € have independent Gaussian distributions with Zero mean and 
covariance matrices having some user-specified structure. Each Yi consists of the 
random coefficients for ith random effect. The variance-covariance matrix structure 
may be different for the different effects. SYSTAT provides the option to specify a 
common covariance parameter for multiple effects, MIXED offers two general 
estimation techniques: ML and REML. Both these methods are iterative. The latter 
produces unbiased estimators. For details of these methods, please refer to Chapter 5: 
Mixed models: Introduction in this volume. 

SYSTAT reports the BLUEs of the fixed effects and BLUPs of the random effects, 
as well as estimates of the variance parameters. 

For each model you fit using MIXED, SYSTAT reports log-likelihood, Akaike 
Information Criterion (AIC), Bayes Information Criterion (BIC), and Akaike 
Information Criterion corrected (AICc) for assessing the fit of the model. 
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Linear Mixed Models in SYSTAT 


Model Estimation (in MIXED) 


To specify a linear mixed model using MIXED, from the menus choose: 


Analyze 
Mixed Models 
Linear Mixed Models... 


Analyze: Mixed Models: Linear Mixed Models 


| Available variable[s]: Dependent 

Intercept [tae e 

ID 
EXERTYPE 
DIET 
PULSE 
TIME 
TIME1 


Dependent. Dependent is the variable you want to examine. Dependent variable should 
be a continuous numeric variable. 
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Fixed effect(s). Select one or more continuous or categorical (grouping) variables 
which you treat as fixed effects. Fixed effects that are not denoted as categorical are 
considered covariates. By default fixed intercept is present in model. If you want 
crossed or nested effects in your model, you need to build these components using 
Cross and Nest buttons. 


Random effect(s). Select one or more continuous or categorical (grouping) variables 

which you treat as random effects. Random effects that are not denoted as categorical 
are considered covariates. If you want interactions or nested effects in your model, you 
need to build these components using Cross and Nest buttons. 


Estimation method. Choose one among the available methods to estimate variance 
components. 


m REML. Uses restricted maximum likelihood method to estimate covariance 
components. It is the default method. 


= ML. Uses maximum likelihood method to estimate covariance components. 


Save. Check the save option to save residuals and other data to a specified data file. The 
following alternatives are available: 


= Marginal residuals. Saves marginal residuals and marginal predicted values. 


= Conditional residuals. Saves conditional residuals and conditional predicted 
values. 


Fixed effect estimates. Saves the estimates of the fixed effects. 
Random effect predictions. Saves the predictions of the random effects. 
Covariance parameters. Saves the estimated covariance parameters. 


Standard errors of fixed effects. Saves standard errors for the fixed effect 
estimates. 


m Residuals/data. Saves marginal and conditional residuals plus all the variables in 
the model. 


m Model. Saves marginal residuals, response variable, and the design matrices. 
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Category 


To specify categorical variables, click the Category tab. Select at least one fixed or 
random effect in Model tab other than intercept to activate this tab. 


Analyze: Mixed Models Linear Mixed Models (2 x) 


| Model] Categor | Random! Options) M im 
| 
| Available vanable(s] — Categorical variable(s) - 
| EXERTYPE | [ EXERTYPE ] 
| TIME | | | 
| TIME1 | | | 
| | | | 
| zi angu 
| [ Missing values 


Dad tame] 


Missing values. Includes a separate category for cases with a missing value for the 


selected variable(s). 
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Random 


To specify covariance structures for random effects and errors, click the Random tab. 


€ 


Error covariance structure 


See 


Initial values for error covariance/cortelation parameter(s]: 


To specify the covariance structures use the following: 
Available effect(s). Available effects are the variables selected as random effects. 


Structure. Choose a random effect and select one of the covariance structures 
available.The available structures are as follows: 


W Variance Components 
= Diagonal 

= Compound Symmetry 
@ Unstructured 
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The default covariance structure is Variance Components. 
Selected effect(s). Effect or effects along with their covariance structures. 


Initial values for random effect covariance parameter(s). Use this option to provide 
initial values for covariance parameters. Specify values for each component in the 
order the effects appear in your model. Separate the values with commas or blanks. Do 
not specify initial values for some of the parameters and leave blanks for others. If you 
do, SYSTAT computes initial values for all covariance components. 


Error covariance structure. Use this option to provide a covariance structure for 
residuals. The available structures are as follows: 

m Variance Components 

= Compound Symmetry 

m AR(1) 

The default covariance structure is Variance Components. 

Initial values for error covariance/correlation parameter(s). Use this option to provide 
initial values for error correlation parameters. Separate the values with commas or 


blanks. Do not specify initial values for some of the parameters and leave blanks for 
others. If you do, SYSTAT computes initial values for all correlation components. 
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Options 


Use the Options tab to specify computational controls for REML or ML method of 
estimation. 


Analyze: Mixed Models: Linear Mixed Models 


| Model | Categoy! Random] Options | 
Convergence options 
Type of convergence: 


Convergence criterion 
O Relative 


Convergence: 

Number of Newton iterations: 
Number of EM iterations: 
Step-halvings: 


Estimation options 
Tolerance: [1e-012 | 


SYSTAT offers the following options for controlling estimation using ML/REML: 
Type of convergence, Check one of the following options to check convergence. 
Three types of convergence checks are available: 


m Hessian. Uses a quadratic form g’ H” g where g is the gradient vector and H is the 
hessian matrix. 


m Likelihood. Uses the difference between log-likelihood at current iteration and the 
log-likelihood at last iteration. 
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W Parameter. Uses maximum of absolute differences between parameter estimates at 
current iteration and parameter estimates at last iteration. 


Convergence criterion. Two criteria are available: 


m Relative. Checks relative difference for convergence. That is, convergence 
checking is done relative to log-likelihood. It is the default option. 


W Absolute. Tests convergence directly against a value specified. 


Convergence. Specify a positive number. SYSTAT stops iterations when convergence 
value is less than this number. 


Number of Newton iterations. Use this to specify maximum number of Newton- 
Rapson iterations for fitting your model. The default is 20. 


Number of EM iterations. Use this to specify maximum number of EM iterations 
before going to Newton-Raphson iterations. Sufficient number of EM iterations 
provide good initial estimates for Newton-Raphson iterations. The default is 5. 


Step-halvings. Use this to specify maximum number of step halvings. The default is 50. 


Tolerance. A check for near singularity. Use Tolerance to guard against singularity 
problems. 

Confidence. Specifies the confidence coefficient for testing purposes. The default is 
0.95. 
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Hypothesis Tests 


To test hypotheses, from the menus choose: 
Analyze 


Mixed Models 
Hypothesis Test... 


Analyze: Mixed Models: Hypothesis Test 


Main |F FE Matice l D Matrix 
Hypothesis: Pairwise > 
Available effects: Selected effect(s) 


EXERTYPE «Required» 
TIME 


TIME1 


E Packs 


E] Tukey's HSD 
C Sidak Cl Hochberg's GT2 


Canin (055 | 


L 
OKK 


You can customize the hypothesis to be tested. Contrasts can be defined across the 
categories of a grouping factor, 


Hypothesis. Select the type of hypothesis. The following choices are available: 
= Pairwise. Compare pairs of groups to determine which pairs differ. 
m Fand R Matrices. Tests the hypotheses corresponding to the F and R Matrices tab. 


The following options are available to compute p-values adjusted for multiple 
comparisons: 
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Bonferroni. Uses Student's t statistic. It sets the family-wise error rate as (- 
Confidence)/ (Total number of comparisons). 

Tukey's HSD. Uses the Studentized range statistic to make all pairwise 
comparisons. This is the default method. 

Fisher's LSD. Equivalent to multiple t tests between all pairs of groups. The 
disadvantage of this test is that no attempt is made to adjust the observed 
significance level for multiple comparisons. 

Scheffé. The significance level of Scheffé's test is designed to allow all possible 
linear combinations of group means to be tested, not just the pairwise comparisons. 
As a result Scheffé's test is more conservative than other tests. 

Sidak. Uses Student's t statistic for pairwise multiple comparisons. 

Hochberg's GT2. Uses Studentized maximum modulus distribution. 


Confidence. Specify the confidence coefficient. The default is 0.95. 
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F and R Matrices 


To specify F and R matrices, select the F and R matrices option of Hypothesis in the 
Mixed Models: Hypothesis Test dialog box. F and R are the matrices of linear weights 
contrasting the coefficient estimates for fixed and random effects respectively. You can 
write your hypothesis in terms of the F and R matrices. 


Analyze: Mixed Models: Hypothesis Test 


[Main | FR Matices | D Mate) — 


Fixed effects): — 
| 


Fixed effect(s). Specify as many numbers as the dimension of your beta vector, In case 
you specify less, SYSTAT takes the unspecified ones as zero; if you specify more, 
SYSTAT ignores the extra ones. 


Random effect(s). Specify as many numbers as dimension of your gamma vector. In 
case you specify less, SYSTAT takes the unspecified ones as zero: if you specify more, 
SYSTAT ignores the extra ones. 
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D Matrix 
D is a null hypothesis vector. By default it is null vector. The D vector, if you use it, 


must have the same number of rows as the F or R matrices. To specify a different D 
Matrix, click the D Matrix tab in the Mixed Models: Hypothesis Test dialog box. 


Analyze: Mixed Models: Hypothesis Test 


Main || F.R Matrices! D Matrix I 


[Use matrix 


Specify a vector of dimension same as the number of rows in F and R matrices. 


Estimate. Check this option for testing significance of contrasts (rows) in F and R 
matrices individually. This test reports estimate of the estimable linear parametric 


function, its standard error and the corresponding t-test. 
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Using Commands 


To analyze linear mixed models using commands first select a data set with USE 
filename and continue with: 


MIXED 
RESET 
CATEGORY grpvarlist / MISS 
MODEL var = INTERCEPT + varlistl 
RANDOM varlist2 / STRUCTURE = VCOMPONENTS or 
CSYMMETRY or DIAGONAL 
or UNSTRUCTURED, 
GROUP = terml MEANS 
REPEATED / STRUCTURE = VCOMPONENTS or CSYMMETRY 
or UNSTRUCTURED 
GROUP = term2 
SAVE filename / MRESIDUALS CRESIDUALS DATA 
COVPARAMETERS FIXED BLUP 
SERRORFIXED MODEL 
ESTIMATE / METHOD = REML or ML TYPE = HESSIAN or 
LIKELIHOOD or PARAMETERS 
CRITERION = RELATIVE or ABSOLUTE 
NEM = nl NNR = n2 CONVERGENCE = dl 
HALF = n3 TOLERANCE = d2 
CONFIDENCE = d4 GSTART=[VECTOR] 
RSTART= [VECTOR] 


To perform hypothesis tests: 


HYPOTHESIS 
PAIRWISE varlist / BONF or LSD or TUKEY or SCHEFFE or 
SIDAK or GT2 
FMATRIX [matrix] 
RMATRIX [matrix] 
DMATRIX [matrix] 
TEST / CONFIDENCE = d5 ESTIMATE 


Usage Considerations 


Types of data. MIXED requires a rectangular data file. 


Print options. MIXED displays covariance parameters and tests of fixed effects for 
PLENGTH SHORT. For PLENGTH MEDIUM, MIXED adds fixed effects estimates. For 
PLENGTH LONG MIXED adds random effects predictions and iteration history. 


Quick Graphs. MIXED produces a Quick Graph of marginal residuals versus marginal 
predicted values. 
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Saving files. Several sets of output can be saved to a file. The actual contents of the 
saved file depend on the analysis. Files may include estimated regression coefficients, 
model variables, residuals, and predicted values. 


BY groups. MIXED analyzes data by groups. 


Case frequencies. MIXED uses the FREQUENCY variable, if present, to duplicate cases. 


Case weights. MIXED uses the values of any WEIGHT variables to weight each case. 


Examples 


Example 1 
From VC to MIXED 


The SYSTAT VC command analyzes variance components models, which constitute a 
special case of mixed effects models. The MIXED command generalizes the VC 
command in one important respect; it allows the covariance matrices of the random 
effects and random errors to be other than multiples of the identity matrix. In other 
words, now the different coefficients under the same random effect can have different 
variances. They may also be correlated. 

While this may appear to be a mild generalization mathematically, it has a deep 
statistical impact. Let us consider the following data set to illustrate this. This example 
will point out how the SYSTAT output for MIXED differs from that of VC. 

An experiment was conducted by students at The Ohio State University in the fall 
of 1993 to explore the relationship between a person's heart rate and the frequency at 
which that person stepped up and down on steps of various heights. The response 
variable, heart rate, was measured in beats per minute. There were two different step 
heights: 5.75 inches (coded as 0), and 1 1.5 inches (coded as 1). There were three rates 
of stepping: 14 steps/min. (coded as 0), 21 steps/min. (coded as 1), and 28 steps/min. 
(coded as 2). This resulted in six possible height/frequency combinations. Each subject 
performed the activity for three minutes. Subjects were kept on pace by the beat of an 
electric metronome. One experimenter counted the subject's pulse for 20 seconds 
before and after each trial. The subject always rested between trials until their heart rate 
returned to close to the beginning rate. Another experimenter kept track of the time 
spent stepping. Each subject was always measured and timed by the same pair of 


experimenters to reduce variability in the experiment. Each pair of experimenters was 
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treated as a block. The data are stored in the SYSTAT data file named HEART. The 
source of the data is the website CMU:DASL (2005). 


Consider the model 


Yin = M * aij + By * Eijk 


where y; is the 1-th observation in the k-th block for the i-th height and j-th frequency. 
Here a; is the combined effect of the i-th height and j-th frequency, and f, is the effect 
of the k-th block. We shall treat the €; 's as fixed. 

Let us see if we should consider BLOCK as a random effect here. In VC, we can do 
so if we consider the experimenters as a random sample from an infinite population of 
experimenters, In other words, in future fresh replications of the experiment, we may 
use other experimenters. In that scenario, however, we must treat the B,'s as 
independently and identically distributed random variables. This necessarily leads to a 
covariance matrix of the formo” I, the form that characterizes variance components 
models. Under what situation then can we treat BLOCK as a random effect, and yet 
have some other covariance structure? 

The answer to this question is the key to understanding when to use the MIXED 
command. Suppose that we do not have any more experimenters at hand, and so all 
future replications must use the same set of experimenters. However, even then we 
should consider the BLOCK effect as random, if we want our model to account for the 
fact that, for a different replication of the same experiment, the same experimenter may 
behave in a slightly different way (say, depending on their mood, which is random.) In 
this case, B, is a random variable for the first block, while B, is a random variable for 
the second block. In this situation the p ‘s may have different variances, even though 
we may still consider them as independent. The VC command is unable to tackle this 
model. So we shall employ the more powerful MIXED command here. 


The input is: 


USE HEART 

LET Y-HR-RESTHR 

MIXED 

CATEGORY HEIGHT FREQUENCY BLOCK 

MODEL Y = INTERCEPT + HEIGHT*FREQUENCY 
RANDOM BLOCK / STRUCTURE - DIAG 
ESTIMATE 
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Dependent Variable : 
Fixed Factor (s) 
Fixed Covariate(s) 
Random Factor (s) 
Estimation Method 


x: 

HEIGHT* FREQUENCY 
Intercept 

BLOCK 


Dimensions 


Covariance Parameters : 
Columns in X 
Columns in 2 
No. of Observations 


Iterations History 


Iteration no. | Iteration type -2L-L 
0 183.775 
À 182.972 
2 181.989 
3 181.652 
4 181.176 
5 180.940 
6 180.810 
ya 180.740 
8 180.701 
9 180.680 
10 180.669 

11 180.663 
12 | NR 180.660 
13 | NR 180.658 
14 | NR 180.657 
15 | NR 180.657 
16 | NR 180.657 
17 | NR 180.657 


Fit Statistics 


Final L-L 3 -90.328 
-2L-L : 180.657 
AIC 1 194.657 
AlC(Corrected) : 201.657 
BIC : 202.903 


Estimates of Covariance Components 


Random Effect | Description Estimate 


BLOCK Variance 1 
Variance 2 0.005 
Variance 3 60.924 
Variance 4 3.794 
Variance 5 0.005 
Variance 6 303.873 


Variance 53.599 
Parameter 


Error variance 


: Residual or Restricted Maximum Likelihood (REML) 


Convergence 


0.006 
0.010 
0.003 
0.005 
0.002 


Linear Mixed Models 
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Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t p-value 
Ai Etica Ua here dec a aS icf vua 
Intercept | 59.231 3.503 5 16.908 0.000 
SCO CADRE ATA c Hic tye neta A SUI 2 Ses 
HEIGHT* FREQUENCY 0*0 i -46.056 4.715 19 -9.768 0.000 
0*1 | -33.688 4.694 19 -7.177 0.000 
0*2 i 724.062 4.720 19 -5.097 0.000 
1*0 | -29.058 4.688 19 -6.199 0.000 
lel į -24.258 4.688 19 -5.175 0.000 
1*2 i 0.000 0.000 . H 


Confidence Intervals of Fixed Effects Estimates 


Effect 


Estimate 


HEIGHT*FREQUENCY 


mede. uen CE 


Estimate Standard 


95.00% Confidence Interval 
Lower 


Confidence Intervals of 


Random Effects Predictors 


95.00% Confidence Interval 


Effect Lower Upper 

| -10.711 718.374 -3.048 

i 0.000 -0.147 0.147 

i -6.989 714.264 0.286 

i 0.851 -2.816 4.518 

H 0.000 -0.149 0.149 

| -17.019 -24.911 79.128 
Type III Tests for Fixed Effects 
Effect | Numerator df Denominator df F-ratio p-value 
HEIGHT*FREQUENCY į 5 19 ^. 20.766 AME 0.000 


Next let us take our analysis one step further by allowing the random BLOCK effects 
to be correlated. This would be a natural model to use, if , for example, the randomness 
of the BLOCK effect is mainly because of certain aspects of the experiment that may 
affect all the experimenters during any replication of the experiment. Such effects may 
be fatigue, or random conditions prevailing during the experiment. However, if we 

assume that all the experimenters are eqaully affected by the common condition then 


11-361 


Linear Mixed Models 


it may be reasonable to assume that the variances are all same and so are the 
covariances between the pairs of distinct blocks. In other words, the covariance matrix 
has a compound symmetry structure. 


The input is: 


MIXED 

CATEGORY HEIGHT FREQUENCY BLOCK 

MODEL Y = INTERCEPT + HEIGHT*FREQUENCY 
RANDOM BLOCK / STRUCTURE = CS 


ESTIMATE 
The output is: 
Fit Statistics 
Final L-L 2 -91.785 
-2L-L : 183.571 
AIC : 189.571 
AlC(Corrected) : 190.771 
BIC : 193.105 


Estimates of Covariance Components 


Random Effect 


BLOCK Variance 59.452 
Parameter 
Compound 4.291 
Symmetry 

Error variance | Variance 57.339 
Parameter 


Ef 


HEIGHT*FREQUENCY 


Confidence Intervals of Fixed Effects Estimates 


i 95.00% Confidence Interval 
E et erates aod. rest 


HEIGHT* FREQUENCY 
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Predictions of Random Effects 


Random Effects Predictors 


95.00* Confidence Interval 


2 4.932 
3 -2.158 
4 i 7.398 
5 i 4.624 
6 i -9.864 


Type III Tests for Fixed Effects 


Numerator df Denominator df F-ratio p-value 


HEIGHT*FREQUENCY | 5 19 19.570 0.000 


In fact, we can take our exploration even further by allowing an arbirary covariance 
structure among the experimenters. There is not much reason to do this for this 
example. Indeed the number of random effect covariance parameters hikes up from a 
modest 2 for CS structure to a staggering 21. The model has too many covariance 
parameters to estimate, which leads to less precise estimation. Yet, being a more 
general model it surely produces smaller residuals. So there is a tradeoff: which one 
should we lay more emphasis on, more precise estimates or smaller residuals? The 
Bayesian Information Criterion (BIC) is designed to resolve this dilemma. It cleverly 


compares both the issues and strikes up a balance, so that better models have smaller 
BIC values. 


Example 2 
Structured Covariance Matrix for Random Errors 


The MIXED command lets the user specify the covariance structure of the random 
errors. This is particularly useful when we have a mixed model in a time series set up, 
as in this example. Here we are interested in the effect of the dose of a drug on the 
growth of rats. We have 5 doses of the drug. Ten rats are assigned to each dose and the 
weight of each of the 50 rats is observed weekly for 11 weeks. Thus, here the data 
come from a designed experiment couched in a time series set up. In such a case it may 
not be a good idea to assume that the random errors are independent. Rather, it is more 
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natural to assume some stationary time series model for them. But before embarking 
upon any formal statistical analysis of the data, let us plot the data. We plot the mean 
body weight against time for the different doses. 


The input is: 
USE RATGROWTH 
DOT WEIGHT * WEEK / GROUP = DOSE OVERLAY LINE YMIN-50 YMAX=150 
The output is: 
Time Series Plot 


Se 
130+ p 4 
P 
ier OA al 
á 27 
= wh Li J| pose 
“A .0 
y 05 
n £2 4 1 
A4 
vs 
when L 4 A— L 
S5 2 4 6 8 10 2 
WEEK 


The plot consists of some nearly parallel straight lines, which leads us to suspect a 
linear dependence on time, free of any interaction with doses. However, the slight 
crossovers between the lines near the two extremes doses not allow us to be sure about 
the absence of the interaction between dose and time. So we use the model 


Yi = &+ By + Eijk 


where i=1,....5,j=1,...,10, and k=1,...,11. Here yj, is the weight of the j-th rat under the 
i-th dose in the k-th week. The rat effect enters through the interaction term Bij- The 
presence of this interaction, which we treat as a random effect, captures the fact that 
the rats constitute a random sample from a large population of rats, and that different 
rats may react to different doses differently. Finally, we model the error as an AR(1) 


process. 
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The input is: 
MIXED 
CATEGORY DOSE WEEK RAT 
MODEL WEIGHT = INTERCEPT + DOSE + WEEK + 
DOSE*WEEK 
RANDOM RAT*DOSE 
REPEATED / STRUCTURE = AR(1) 
ESTIMATE 

The output is: 

Fit Statistics 

Final L-L : -2726.998 

-2L-L : 5453.996 

AIC : 5459.996 

AIC (Corrected) : 5460.044 

BIC : 5472.609 


Estimates of Covariance Components 


Random Effect Description Estimate 


i 
+ 
! Variance 182.892 
| Parameter 
€ anana: p as nh a T 
Error variance | Variance 2623.384 
| Parameter 
| Error -0.008 
| Correlation 
| (ARQ)) 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t p-value 
PL SIPC S Ue cbr 8 + EM PRONUS fie ipd 
DOSE*WEEK 0*1 i 450 4.142 0.000 
0*2 i 450 4.513 0.000 
0*3 ‘ 450 4,847 0.000 
0*4 1 450 5.319 0.000 
0*5 1 450 5.671 0.000 
0*6 i 450 6.226 0.000 
0*7 i 450 6.596 0.000 
0*8 i 450 6.978 0.000 
0*9 ' 450 7.462 0.000 
0*10 H 450 7.933 0.000 
0*11 i 450 8.447 0.000 
0.5*1 | 450 4.233 0.000 
0.5*2  ] 450 4.591 0.000 
0.5*3 | 450 4.937 0.000 
0.5*4 | 450 5,211 0.000 
0.5*5 | 450 5.659 0.000 
0.5*6 | 450 6.041 0.000 
0.5*7 | 450 6.525 0.000 
0.5*8 | 450 6.948 0.000 
0.5*9 | 450 7.325 0.000 
0.5*10 | 450 7.963 0.000 
0.5*11 ; 450 8.369 0.000 
13] 450 — 3.969 0.000 
12 —| 450 — 4.352 0.000 
r3 of 450 — 4.692 0.000 
1*4 i 450 4.984 0.000 
1*5 : E y 450 5.444 0.000 
1*6 | 96.600 16.752 450 5.766 0.000 
1*7 ! 106.000 16.752 450 6.328 0.000 
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1*8 
1*9 
1*10 
TALE 
4*1 
4*2 
4*3 
4*4 
4*5 
4*6 
4*7 
4*8 
4*9 
4*10 
4*11 
8*1 
8*2 
8*3 
8*4 
B*5 
8*6 
8*7 
8*8 
8*9 
8*10 
8*11 


i 
1 
i 
1 
i 
1 
1 
i 
4 
1 
i 
i 
1 


117.700 
124.703 


16.752 450 6.668 
16.752 450 7.205 
16.752 450 14.822 
16.752 450 8.077 
16.752 450 4.000 
16.752 450 4.262 
16.752 450 4.614 
16.752 450 4.955 
16.752 450 5.253 
16.752 450 5.605 
16.752 450 6.041 
16.752 450 6.429 
16.752 450 6.817 
16.752 450 7.205 
16.752 450 7.593 
16.752 450 3.797 
16.752 450 4.089 
16.752 450 4.471 
16.752 450 4.811 
16.752 450 5.092 
16.752 450 5.486 
16.752 450 5.766 
16.752 450 6.220 
16.752 450 6.548 
16.752 . 450 7.026 
16.752 450 7.444 
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All the coefficients are significant, as judged by the small p-values.: 


Predictions of Random Effects 


Effect 


RAT*DOSE 


1 | Estimate 


Standard Error df 


t 


p-value 
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37*4 10.531 450 0.523 0.601 
38*4 10.531 450 -0.610 0.542 
39*4 10.531 450 -0.112 0.911 
40*4 10.530 450 -0.029 0.977 
41*8 . 10.530 450 -0.199 0.842 
42*8 1.084 10.531 450 0.103 0.918 
43*8 0.489 10.531 450 0.046 0.963 
44*8 H 2.280 10.531 450 0.217 0.829 
45*8 ; -7.066 10.531 450 -0.671 0.503 
46*8 1 8.323 10.531 450 0.790 0.430 
47*8 1 10.354 10.531 450 0.983 0.326 
48*8 | 711.750 10.531 450 -1.116 0.265 
49*8 1 -3.853 10.531 450 -0.366 0.715 
50*8 i 2.234 10.530 450 0.212 0.832 


Almost none of the random coefficients is significant. The only exception is the 
interaction between rat 28 and dose 1. 


Example 3 
Repeated Measures Experiment with Covariates 


In this example we are interested in analyzing the effect of diet and exercise on pulse 
rates. We shall consider two types of diet: low-fat and not low-fat; and three types of 
exercises: at rest, walking leisurely and running. In each of the 6 cells of the resulting 
two-way layout we assign 5 persons. For each person we measure the pulse rate at 4 
different time points. The time points vary from person to person. However, the first 
measurements are made simultaneously and correspond to time=0. 


Low-fat Not low-fat 


npbi i His Hd 


a 
| 
| 
N 
ziN 
E 
m NI 
>o 
>o 
>o 
>o 
>o 
>o 
>o 
>o 
>o 
N 
E 


MUTET 
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Let us first plot the pulse rates over time for each of the individuals. The plots suggest 
that the curves are quadratic in nature. So we shall try to fit a quadratic regression of 
pulse rate over time: 


pe 2 
Yin = Ajj + Pijtia * Yijt ijkl * &ija 


where y;;,, is the pulse rate at the k-th time point for the 1-th person under the 

(i, j)-th diet-exercise regime. Here i=1,2, j=1,2,3, k=1,...,4 and 1=1,...,5. Notice that we 
have allowed the coefficients of the quadratic regression in the 6 cells to depend on the 
cell. In other words, the coefficients may depend on the particular diet-exercise regime. 
Now, the diets are qualitatively specified only by their fat contents. So it may be 
reasonable to assume that the diets used in the experiment are random samples from a 
large population of diets with the specified fat level. So we further model each of the 
coefficients j, Bij and y; as a fixed exercise effect plus a random contribution from 
the diet. This is an example of multi-level modelling, where the first level consists of 
modeling the pulse rates in terms of time, and the second level consists of modeling the 


coefficients in terms of diet and exercise, as shown below. 


Qi a+bj+pi 


Bj = c+dj+qi 

yj = e*fj*ri 
Here pj, q; and r; are the random diet effects in the second level model equations. 
Combining the two levels, we get the final model 

yia = (a+bj) + tip dj tijk) + (e ti^ fj tig) + (pi + qiti + tig) ijk 


Collecting similar terms we see that the right hand side of the model consists of an 


intercept, the main exercise effect, main time effects (linear and quadratic), and 


interactions between the times effects and exercise effect. 
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The input is: 


USE EXER 
MIXED 
CATEGORY ID EXERTYPE DIET 
MODEL PULSE = INTERCEPT + EXERTYPE + TIME, 
+ TIME*EXERTYPE + TIME*TIME, 
+ TIME*TIME*EXERTYPE 
RANDOM INTERCEPT TIME TIME*TIME/GROUP = DIET 
ESTIMATE 


Notice the GROUP option in the RANDOM line. This fits different random intercepts, 
linear and quadratic terms for different diets. 


The output is: 

Fit Statistics 

Final L-L : -429.200 
-2L-L * 858.400 
AIC : 866.400 
AIC(Corrected) : 866.784 
BIC : 877.165 


Estimates of Covariance Components 


Variance 12.296 
Parameter 


Variance 0.000 
Parameter 


TIME* TIME | Variance 0.000 
| Parameter 

M incdus oS en ee 

Error variance | Variance 57.724 
| Parameter 


Estimates of Fixed Effects 


Effect Level Standard Error df t p-value 


284478.623 
284478.623 
284478.623 


TIME*TIME*EXERTYPE TIME*TIME*1 | . 0.000 r 
TIME*TIME*2 . 0.000 4 
TIME*TIME*3 ; . 0.000 " 


Confidence Intervals of Fixed Effects Estimates 
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95.00% Confidence Interval 
Estimate Lower Upper 


Effect Level 


564131.372 
-564131.355 564132.264 
-564131.346 564132.272 
-564131.240 564132.379 


TIME*TIME*EXERTYPE TIME*TIME*1 
TIME*TIME*2 | 
TIME*TIME*3 | 


Predictions of Random Effects 


Group Group Level Effect 


1 Intercept 
DIET 2 Intercept 


DIET 1 TIME*TIME 
DIET 2 TIME*TIME 


Confidence Intervals of Random Effects Predictors 


95.00% Confidence Interval 
Lower Upp“ 


Group Group Level Effect 


Intercept 
Intercept | 


i 
+ 
1 


DIET 1 TIME*TIME 0.000 0.000 . 

DIET 2 TIME*TIME 0.000 0.000 0.000 
Type III Tests for Fixed Effects 
Effect | Numerator df Denominator df F-ratio p-value 

O A E ESOL BE? 

EXERTYPE 2 104 0,597 0.552 
TIME a 1 2 0.000 1.000 
TIME*EXERTYPE 3 104 9.182 0.000 
TIME*TIME 1 2 0.000 . 
TIME*TIME*EXERTYPE 3 104 3.144 0.028 

Example 4 


Estimation: ML and REML 


SYSTAT MIXED allows two different methods to estimate the covariance parameters: 
Maximum Likelihood (ML) and Residual/Restricted Maximum Likelihood (REML). 
The more popular choice is REML, which is the default in SYSTAT. For large sample 
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sizes, the two methods give comparable estimates. The main objection against ML 
estimators is that they are biased, while REML estimators are unbiased. This data set 
from Brownlee (1960) pertains to bacteriological testing of milk. Twelve milk samples 
were tested in all 6 combinations of 2 types of bottles and 3 types of tubes. Ten tests 
were run on each combination and the response was the number of positive tests in 
each set of ten. 


The input is: 


USE MILK 

MIXED 

CATEGORY TUBES BOTTLES 

MODEL Y = INTERCEPT + TUBE$ + BOTTLES + TUBE$*BOTTLE$ 
REPEATED / GROUP = SAMPLE 

ESTIMATE / METHOD = REML 


The output is: 

Fit Statistics 

Final L-L : -131.888 
-2L-L : 263.775 
AIC : 265.775 
AlC(Corrected) : 265.838 
BIC : 267.965 


Estimates of Covariance Components 
Random Effect 


Error variance 


Description Estimate 


Variance 2.542 
Parameter 


cU Tw 


Notice that the estimated error variance is 2.542. We shall compare this with the 
estimate obtained by the ML method later. 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t  p-value 
poen —€— € (lh 
Intercept 2.500 0.460 66 5.432 0.000 
TUBES A -1.250 .651 66 -1.921 F “0.059 
B i -0.167 0.651 66 -0.256 0.799 
c i 0.000 0.000 
— A 8 
BOTTLES I i 0.167 0.651 66 0.256 0.799 
II i 0.000 0.000 
e M A +--+ +--+ 22+ 
TUBES *BOTTLES A*I i 0.333 0.920 66 0.362 0.718 
A*II | 0.000 0.000 
B*I 70.417 0.920 66 70.453 0.652 
B*II 0.000 0.000 
C*I i 0.000 0.000 
C*II "i 0.000 0.000 
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Observe that the effect of Tube A is not significant at 0.05 level (since its p-value is 
more than 0.05). This inference will change when we use ML later. 


Type III Tests for Fixed Effects 


Effect | Numerator df F-ratio p-value 
TUBES i 2 66 2.858 0.065 
BOTTLES i 1 66 0.137 0.713 
TUBES*BOTTLES | 2 66 0.333 0.718 


Next we shall apply ML estimation with the same model. 


The input is: 


USE MILK 

MIXED 

CATEGORY TUBE$ BOTTLES 

MODEL Y = INTERCEPT + TUBES + BOTTLES + TUBE$*BOTTLE$ 
REPEATED / GROUP = SAMPLE 

ESTIMATE / METHOD = ML 


The output is: 


Estimates of Covariance Components 


Random Effect | Description E 


cree ew ew a mam ana exce dc E t d 


Error variance | Variance 
| Parameter 


Now the estimated error variance is 2.330, instead of 2.542 as was obtained by the 
REML method. This is because the REML method adjusts the denominator degrees of 
freedom to account for the fixed effects. The corrected number of degrees of freedom 
is less than the raw degrees of freedom used by the ML method. That is why the 
variance estimate obtained by REML is larger than that obtained by the ML method. 


Estimates of Fixed Effects 
' Estimate Standard Error 


p-value 


TUBES*BOTTLES A*I 1 0.333 . 
A‘ II | 0.000 0.000 
B*I f -0.417 0.881 66 -0.473 0.638 
Beil 0.000 0.000 
c*1 $ 0.000 0.000 
C*II 0.000 0.000 
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This table looks similar to what we had using REML estimation earlier. This is because 
REML is only a little variation of ML. However, the coefficient for Tube A is now 
significant at 0.05 level (p-value less than 0.05). When we used REML method the 
same coefficient was not significant at 0.05 level. 


Example 5 
Hypothesis testing 


The data set used here is adapted from Milliken and Johnson (1992). Here we are 
comparing four different paints. The paints are of two different colors and are 
manufactured by two different companies. We shall call them Yellow], Yellow2, 
Whitel and White2, where the 1 and 2 refer to the company. Each paint is applied on 
three different paving surfaces: Asphalt, Asphalt2, and Concrete. The response is the 
life-time measured in weeks. Milliken and Johnson have reported only the cell means 
and the error sum of squares. The data set has been generated artificially to have the 
same cell means and error sum of squares as the original data. 

We shall fit the following model to this data set: 


Yik = ttj t 0 + B + ij + Eig 


where i-1,...,4, j-1,2,3 and k=1,2,3. Here y;j, is the k-th measurement of the life time 
ofthe i-th paint as applied on the j-th surface. In this model we shall treat all the effects 
as fixed. Later we shall see the change introduced by considering some of the eefects 
as random. 


The input is: 


USE PAINTS 

MIXED 

CATEGORY PAINT$ PAVE$ 

MODEL Y = INTERCEPT + PAINTS + PAVES + PAINTS$*PAVES 
ESTIMATE/METHOD = REML 
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The output is: 
Fit Statistics 
Final L-L : -75.955 
-2L-L : 151.910 
AIC : 153.910 
AIC(Corrected) : 154.092 
BIC : 155.088 
Estimates of Covariance Components 
Random Effect | Description Estimate 
- * 
Error variance | Variance 18.961 

| Parameter 
Estimates of Fixed Effects 

Standard Error df t p-value 


Whitel 
White2 

Yellowl 
Yellow2 


Asphalt1 
Asphalt2 
Concrete 


PAINTS * PAVES Whitel*Asphaltl 


Whitel*Asphalt2 

Whitel*Concrete 

White2*Asphaltl 

White2*Asphalt2 

White2*Concrete . 

Yellowl*Asphaltl 5.028 24 -4.773 
Yellowl*Asphalt2 5.028 24 -4.972 0.000 
Yellowl*Concrete 0.000 

Yellow2*Asphaltl 0.000 

Yellow2*Asphalt2 0.000 . 
Yellow2*Concrete | 0.000 


Type III Tests for Fixed Effects 


Numerator df Denominator df 


Effect i 
A E O 
PAINTS | 3 24 
PAVES t 2 24 
PAINTS*PAVES } 6 24 


Now, a number of hypotheses are of interest. We may be interested in knowing if the 
Yellow! paint differs significantly from Yellow2. This corresponds to testing equality 
of the expectations of the total of the Yellow! observations and the total of the Yellow2 


observations. So we have the hypothesis 


H:304, - 302 + (ri * Yi + 3) - (Yar + 12 *yn) = 0 
We can test this hypothesis in SYSTAT as follows: 
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The input is: 
HYPOTHESIS 
FMATRIX [0, 
3.-3..0 0% 
0 0 0, 
1 X G 
-1 -1 -1, 
0 0 o, 
o 0 oj 
TEST 


To understand the F matrix, imagine all the parameters listed in a row in the same order 
as given in the MODEL line: 


Q4, OL, Q5, 04, Bi, Bo, Bs, is Yos Ys Yo Ya2s Y23» Y31> Y32» Ys "ats Yaz» Yas 
Then, the F matrix is obtained by listing all the null hypothesis coefficients for these 
parameters. 

The output is: 


F Matrix 


0.000 3.000 -3.000 0.000 0.000 


0.000 0.000 1.000 1.000 1.000 71.000 


-1.000  -1.000 0.000 0.000 0.000 0.000 


0.000 0.000 


F-ratio Test 


1.000 ; 24 8.575 0.007 


No abbreviation is allowed here. The right-hand side of our Hp is zero. So you could 
also write the following: 
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The input is: 

HYPOTHESIS 

FMATRIX [0, 
1.5 -1.5 00, 
O 0 9, 
0.5 0.5 0.5, 
-0.5 -0.5 -0.5, 
0 0 0; 
0 0* 0] 

TEST 


because the constant 0.5 just factors out of Hp. 


The output is: 
F Matrix 
1 2 3 4 5 6 
0.000 1.500  -1.800 0.000 0.000 0.000 
7 8 9 10 1 12 


0.000 0.000 0.500 0,500 0.500 -0.500 


-0.500  -0.500 0.000 0.000 0.000 


0.000 0.000 


F-ratio Test 


Numerator df ; 


Next let us test if the life-time of the Yellow paints differs 


The input is: 

HYPOTHESIS 

FMATRIX [0, 
33-3 -3, 
000, 
d etii 
iX 
E 
-1 -1 -1] 


TEST 


p-value 


24 8.575 0.007 


Linear Mixed Models 


from that of the White paints. 
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The interaction terms are taken care of by SYSTAT to preserve estimability as 
mentioned earlier. 


The output is: 


F Matrix 


1.000 1.000 


.000 -1.000 


F-ratio Test 


Numerator df Denominator df F-ratio p-value 


Testing between the two types of asphalt can be achieved as follows. 


The input is 

HYPOTHESIS 

FMATRIX [0, 
0000, 
4 -4, 0, 
1-10, 
1-10, 
1-10, 
1 -1 0] 

TEST 


Here the first five 0's keep the u and &;'s out of the picture. 


The output is: 


F Matrix 


0.000 0.000 0.000 0.000 0.000 4.000 


-4.000 0.000 1.000 -1.000 0.000 1.000 
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F-ratio Test 


Denominator df F-ratio p-value 


Numerator df 


ae DR 


So far we have been treating all the effects as fixed. Now let us see what happens if we 
let the paint effect and the interaction be random. This will be the case, for instance, if 
the different cans of paints of the same color from the same company show 
significantly different life-times. 


The input is: 


USE PAINTS 

MIXED 

CATEGORY PAINT$ PAVE$ 

MODEL Y = INTERCEPT + PAVES 
RANDOM PAINTS + PAVE$*PAINTS 
ESTIMATE 


The output is: 


Estimates of Covariance Components 


Description Estimate 


+ 
PAINTS | Variance 21.389 
| Parameter 
A A a oe 
PAVES*PAINTS | Variance 29.313 
| Parameter 
———— Hilden ERES 
Error variance | Variance 18.961 
| Parameter 


Estimates of Fixed Effects 


Effect Level | Estimate df t p-value 
pin abd Nae M be RT 4 A AA eu Hin 
Intercept } 
dac s Coo ira Ai use OE + 
PAVE: Asphaltl i -2.758 4.221 6 e 
$ Asphalt2 H -1.758 4.221 6 -0.417 0.691 
Concrete | 0.000 0.000 H 5 
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Predictions of Random Effects 


Effect Effect Level | Estimate Standard Error df t p-value 
icio A apt cha APIS omisi Secs a DA, ARA Metti AI. IMA 
PAINTS Whitel i 3.328 24 0.241 0.812 
White2 i 3.328 24 1.402 0.174 
Yellowl i 3.328 24 -1.240 0.227 
Yellow2 i 3.328 24 -0.403 0.690 
PAVE$*PAINT$ Asphaltl*Whitel 3.886 24 0.571 0.573 
Asphaltl*White2 3.886 24 0.600 0.554 
Asphaltl*Yellowl 3.886 24  -1.561 0.132 
Asphaltl*Yellow2 3.886 24 0.390 0.700 
Asphalt2*Whitel 3.886 24 -0.064 0.950 
Asphalt2*White2 ` 3.886 24 0.600 0.554 
Asphalt2*Yellowl | -5.242 3.886 24 -1.349 0.190 
Asphalt2*Yellow2 | 3.160 3.886 24 0.813 0.424 
Concrete*Whitel | -0.872 3.886 24 -0.224 0.824 
Concrete*White2 | 1.734 3.886 24 0.446 0.659 
Concrete*Yellowl | 5.651 3.886 24 1.454 0,159 
Concrete*Yellow2 | 76.513 3.886 24 -1.676 0.107 


Now let us compare the two types of asphalt again. 


The input is: 


HYPOTHESIS 
FMATRIX [0, 

1 -1 0] 
TEST 


The output is: 
F Matrix 


F-ratio Test 


Numerator df lue 


0.814 


Observe how the output differs this time. Here we are using the so-called broad 
inference space, where the random effects are left unspecified. If we want to perform 
the same comparison but with the effects of the yellow paints held fixed at their 
currently predicted values, (i.e., if we plan to replicate the experiment with fresh 


supplies of the white paints, but still use yellow paints of the old standard) then we can 
do the following: 
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The input is: 
HYPOTHESIS 
FMATRIX [0, 

1 -1 0] 
RMATRIX [3 3 0 0, 

ie 

i 1.1, 

0 0 0, 

0 0 0] 
TEST 


This is an example of a test performed in intermediate inference space. 


The output is: 


F Matrix 


0.000 1.000 -1.000 0.000 


R Matrix 


F-ratio Test 


Numerator df | Denominator df ^ F-ratio p-value 


Broad inference spaces answer most of the needs that occur in real life. One must be 
very careful to interpret tests performed in other inference spaces. 


Example 6 
Post hoc tests 


The data set used here is adapted from Hand et al. (1994). Data were collected on the 
genus of flea beetle Chaetocnema, which contains three species: concinna (Con), 

heikertingeri (Hei), and heptapotamica (Hep). Measurements were made on the width 
and angle of the aedeagus of 74 beetles. The goal of the original study was to form a 
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classification rule to distinguish the three species. Here we shall analyze if angle has 
enough information to distinguish among the three classes. First, we fit a one-way 
ANOVA model. 


The input is: 


USE FLEABEETLE 

MIXED 

CATEGORY SPECIES$ 

MODEL ANGLE = INTERCEPT + SPECIES$ 
ESTIMATE/METHOD = REML 


The output is: 

Fit Statistics 

Final L-L : -106.033 
-2L-L : 212.066 
AIC : 214.066 
AlC(Corrected) : 214.124 
BIC : 216.329 


Estimates of Covariance Components 


Random Effect į Description Estimate 

e A AI 

Error variance | Variance 1.014 
| Parameter 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t p-value 


Confidence Intervals of Fixed Effects Estimates 


95.00$ Confidence Interval 
Li 


SPECIES$ 


Type III Tests for Fixed Effects 


Ef | Numerator df Denominator df F-ratio p-value 


SPECIES$ | 


Then we test if all the coefficients are the same or not. If this null hypothesis gets 
accepted, then there is not much hope for us. 
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The input is: 
HYPOTHESIS 


FMATRIX [0 1 -1 0; 01 0 -1] 
TEST 


The output is: 


F Matrix 


F-ratio Test 
Denominator df F-ratio p-value 


"1 129.633 0.000 


Numerator df 


Notice that the p-value is small and so we can safely reject the null hypothesis. But all 
that this test tells us is that not all the coefficients are the same. We need to test 
something stronger: whether all the coefficients are distinct. For this we carry out two 
pairwise tests using Scheffé's method. 


The input is: 


HYPOTHESIS 
PAIRWISE SPECIES$ / SCHEFFE 
TEST 


The output is: 


Least squares means for effect SPECIES$ 
95.00% Confidence Interval 


Level | Estimate Standard Error df Upper 
prid sint Hillside e E 

Con i 14.095 0.220 71 

Het i 14.290 0.181 71 

Hep 10.091 0.215 "n 


Scheffe Test of effect SPECIES$ 


SPECIESS  SPECIESS | Difference andard Error t p-value Lower Upp 
Gon — mei 0.285  -0.685 0.791 i 0.5 
0.307 13.033 0.000 3.236 4.7 


0.281 14.958 0.000 3.497 4.9 


11-382 


Chapter 7 


Example 7 
Fine Tuning 


In any iterative method there are a number of tuning parameters whose values need to 
be specified. These include initial values, error tolerance and maximum number of 
iterations allowed. However, SYSTAT hides most of the details from the user by 
specifying clever defaults. For instance, before starting the iterations, SYSTAT solves 
the problem approximately by some simple noniterative method, and then uses the 
approximate answer as the initial value for the iterative algorithm. However, there may 
be a rare situation where the user has a better initial value to suggest than what 
SYSTAT uses by default. In such a case, SYSTAT lets the user override the default. 
This is an advanced feature, which you will hardly need for most real life data sets. 
Indeed, we shall use a synthetic data set to illustrate the use of fine tuning. 


Yik ~ Bo; pj ei 


where i=1,...,5, j=1,2,3 and k-1,...,100. We take p and as fixed, a, and B; as random, 
having independent N(0,1) distribution. The random errors, € ijk'S are assumed 
independently distributed as N(0,0.5). We simulate this data set, and estimate the 
parameters. 


The input is: 


USE SIMUL1 

MIXED 

CATEGORY I J 

MODEL Y = INTERCEPT + I 


RANDOM J 

ESTIMATE 
The output is: 
Estimates of Covariance Components 
Random Effect ¡ Description Estimate 
SRE iia e PE A A ed 
J i 1.240 
e, RN ii A S pu a 
Error variance | Variance 0.502 


| Parameter 


Now suppose that some more data are collected. We simulate this fresh data from the 
same model. The new data set is stored in S/MUL2. Rather than analyzing S/MUL2 
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from the scratch, we can specify the estimates from the last analysis as initial values. 
But first let us see what happens if we do not supply the initial values. 


The input is: 


USE SIMUL2 

MIXED 

CATEGORY I J 

MODEL Y = INTERCEPT + I 
RANDOM J 

ESTIMATE 


The output is: 


Iterations History 


Iteration -2L-L Convergence 


Fit Statistics 


Final L-L y -219.371 
-2L-L : 438.741 
AIC : 442.741 
AlC(Corrected) : 442.804 
BIC : 449.287 


Estimates of Covariance Components 


Description Estimate 


+ 

| Variance 1.192 
| Parameter 
* 


Variance 0.491 
Parameter 


Next let us investigate the effect of supplying initial values. 


The input is: 


USE SIMUL2 

MIXED 

CATEGORY I J 

MODEL Y = INTERCEPT + I 


RANDOM J 
ESTIMATE / GSTART - [1.240 0.502] 
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The output is: 
Iterations History 
It Convergence 
0.000 
0.000 
0.000 
0.000 
0.000 
Fit Statistics 
Final L-L 3 -219.371 
-2L-L : 438.741 
AIC : 442.741 
AlC(Corrected) : 442.804 
BIC : 449,287 
Estimates of Covariance Components 
Random Effect | Description Estimate 
x. mdi riii d Hiscan i co cd baie ae 
J Variance 1.192 
arameter 
Error variance | Variance 0.491 
| Parameter 
Notice how the number of iterations have gone down. This is because now the 
iterations have started already close to the answer. While the advantage in terms of 
lower computing time requirement may not be important for a high speed computer, 
this feature may prove useful for online data sets that keep on coming at a high rate. 
Then this feature may be used to update existing estimators. 
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Hierarchical Linear Mixed Models 


Arnab Chakraborty and Ravindra Jore 


Hierarchical Linear Mixed Models (HLMM) fits and analyzes mixed models with 
structured covariance/correlation matrices for random effects and residuals. As in 
LMM, HLMM also provides Variance Components, Compound Symmetry, Diagonal, 
and Unstructured as random effects covariance structures. As error covariance 
structures HLMM provides Variance Components, Compound Symmetry, and Auto- 
Regressive(1). You can fit various models like random intercept model, random 
coefficients model, variance components model, mixed effects ANOVA model, 
growth-curve model, and models with autocorrelated errors using HLMM. HLMM 
allows random effects to be both categorical and continuous. 

In HLMM, SYSTAT provides two methods to estimate covariance parameters, 
viz., Maximum Likelihood (ML) and Restricted/Residual Maximum Likelihood 
(REML). SYSTAT provides the following as default output: 

W covariance parameter estimates. 

m fixed effect estimates and random effect predictions along with their standard 
errors, confidence intervals, and t-tests for testing the significance. 
F-ratio tests for fixed effects. 


log-likelihood, Akaike Information Criterion (AIC), Akaike Information 
Criterion Corrected (AICc), Bayesian Information Criterion (BIC) and iteration 


history. 
HLMM provides save options to save residuals, predictions, model parameter 
estimates with their standard errors, and other statistics to a new data file you specify. 
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Statistical Background 


A general linear mixed model is a model of the form 


y = XB+Zi1+...+Z 19 +48 


where y is the data vector, X and Z;'s are known matrices (either design matrices or 
covariate matrices), D is the vector of fixed effects, each y, is a vector of random 
effects, and is the random error vector. Here y is a random vector, whose randomness 
comes partly from the random vector y and partly from € . We assume that the random 
vectors y, and e have independent Gaussian distributions with zero mean and variance 
matrices having some user-specified structure. Here each y, consists of the random 
coefficients for one random effect. The variance-covariance matrix structure may be 
different for the different effects. SYSTAT provides the option to specify common 
covariance parameters for multiple effects. MIXED offers two general estimation 
techniques: ML and REML. ML method finds the parameter estimates such that -2 log- 
likelihood is minimum. ML method reports biased estimates since it does not account 
for the degrees of freedom for the estimation of fixed effects estimates. REML produces 
unbiased parameter estimates. Both these methods are iterative. The latter produces 
unbiased estimates. SYSTAT reports the Best Linear Unbiased Estimates (BLUEs) of 
the fixed effects and Best Linear Unbiased Predictors (BLUPs) of the random effects, 
as well as estimates of the variance parameters. BLUEs and BLUPs are accompanied 
by their estimated standard errors, two-sided 9594 confidence intervals, and test of 
significance. 

For each model you fit using MIXED, SYSTAT reports log-likelihood, Akaike 
Information Criterion (AIC), Bayesian Information Criterion (BIC), and Akaike 
Information Criterion Corrected (AICc) for assessing the fit of the model. 


11-387 


Hierarchical Linear Mixed Models 


Hierarchical Linear Mixed Models in SYSTAT 


Model Estimation (in MIXED) 


To fit a hierarchical linear mixed model using SYSTAT, from the menus choose: 


Analyze 
Mixed Models 
Hierarchical Linear Mixed Models... 


'@ Analyze: Mixed Models: Hierarchical Linear Mixed Models 


Available vañable(s) 
Intercept zi 
RATIO 
SPECTROMTR$ 


Dependent. Dependent is the variable you want to model. The dependent variable 
should be continuous and numeric. 


Fixed effect(s). Select one or more continuous or categorical variables which you treat 


as fixed effects. Fixed effects that are not denoted as categorical are considered 


covariates. If you want crossed or nested effects in your model, you need to build these 


components using Cross and Nest buttons. 
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Random effect(s). Select one or more continuous or categorical variables which you 
treat as random effects. Random effects that are not denoted as categorical are 
considered covariates. If you want interactions or nested effects in your model, you 
need to build these components using Cross and Nest buttons. An effect can be fixed 
as well as random. 


Estimation method. Choose one among the available methods to estimate variance 
components. 


= REML. Uses restricted maximum likelihood method to estimate covariance 
parameters. It is the default method. 


= ML, Uses maximum likelihood method to estimate covariance parameters. 


Save. Check the save option to save residuals and other data to a new data file. The 
following alternatives are available: 


= Marginal residuals. Saves marginal residuals and marginal predicted values. 


m Conditional residuals. Saves conditional residuals and conditional predicted 
values. 


m Fixed effect estimates. Saves the estimates of the fixed effects. 

= Random effect predictions. Saves the predictions of the random effects. 
= Covariance parameters. Saves the estimated covariance parameters. 
" 


Standard errors of fixed effects. Saves standard errors for the fixed effect 
estimates. 


= Residuals/data. Saves marginal and conditional residuals plus all the variables in 
the working data file. 


m Model. Saves marginal residuals, response variable, and the design matrices. 
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Category 


To specify categorical variables, click the Category tab. Select at least one fixed or 
random effect in Model tab other than intercept to activate this tab. 


SPECTROMTRS 
PLOT 


Missing values. Includes a separate category for cases with a missing value for the 
selected variable(s). 
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Random 


To specify covariance structures for random effects and errors, click the Random tab. 


Mixed Models: Hierarchical Linear Mixed Models 


Initial values. 


Random effect covariance parameter(s) 


Random effect. Random effect column lists all the effects denoted as random effects. 
Error is also listed. 


Subject. Specify a subject effect to define hierarchical structure in random effects. 
Subject defines a block diagonal structure in random effects covariance matrix. That 
is, the covariance between two subjects is zero. You can define subject effect for errors 


also. 


Covariance structure. For a random effect select one of the covariance structures 
available, viz., Variance components, Diagonal, Compound symmetry, or 
Unstructured to specify as its covariance structure. For errors, select one of the 
covariance structures, viz., Variance components, Compound symmetry, or AR(1). 
The default structure is Variance components. 
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Random effect covariance parameter(s). Use this option to provide initial values for 
covariance parameters of random effects. Specify values for each component in the 
order the effects appear in your model. Separate the values with commas or blanks. 
You cannot specify initial values for some parameters and leave others blank. Anyhow, 
SYSTAT computes initial values for all covariance components if you do not specify 
some/all values. Specify initial values that satisfy parameters constraints. Initial values 
of parameters should be such that the variance-covariance matrix of random effects 
should be at least positive semi-definite. 


Error covariance/correlation parameter(s). Use this option to provide initial values 
for correlation parameters. Separate the values with commas or blanks. Make sure that 
the initial values construct positive-definite error covariance matrix. 


Other Subject(s) 


Select Other from a subject drop-down list to select other subjects for your random 
effects. Selecting Other pops up Other Subject(s) dialog box. 


Other Subject(s) 


SPECTROMTRS 
PLOT 


Use Add, Cross, and Nest buttons to build the subjects. 
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Options 


Use Options tab to specify computational controls for ML or REML method of 
estimation. 


Number of Newton iterations: 
Number of EM iterations: 
Step-halvings: 

Estimation options 


Tolerance: |1e-012 


SYSTAT offers the following options for controlling estimation using ML/REML: 
Type of convergence. Check one of the following options to check convergence. 
Three types of convergence checks are available: 


m Hessian. Uses a quadratic form g' H ! g where g is the gradient vector and H is the 
hessian matrix. 


m Likelihood. Uses the difference between log-likelihood at current iteration and the 
log-likelihood at last iteration. 


W Parameter. Uses maximum of absolute differences between parameter estimates at 
current iteration and parameter estimates at last iteration. 


Convergence criterion. Two criteria are available: 
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m Relative. Checks relative difference for convergence. That is, convergence 
checking is done relative to log-likelihood. It is the default option. 


= Absolute. Tests convergence directly against a value specified. 


Convergence. Specify a positive number. MIXED stops iterations when convergence 
value is less than this number. 


Number of Newton iterations. Use this to specify maximum number of Newton- 
Rapson iterations for fitting your model. The default is 20. 


Number of EM iterations. Use this to specify maximum number of EM iterations 
before going to Newton-Raphson iterations. Sufficient number of EM iterations 
provide good starting estimates for Newton-Raphson iterations. The default is 5. 


Step-halvings. Use this to specify maximum number of step halvings. The default is 50. 
Tolerance. A check for near singularity. Use Tolerance to guard against this singularity 
problem. 


Confidence. Specify the confidence coefficient for testing purposes. The default is 
0.95. 
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Hypothesis Test 


To test hypotheses, from the menus choose: 


Analyze 
Mixed Models 
Hypothesis Test... 


Analyze: Mixed Models: Hypothesis Test 
Main [F B Matices ED Ma 


Hypothesis: [Paiwise B 
Available effects: Selected effects} 


SPECTROMTR$ «Required» 
is ual 
Ce ] 


Moss eX 
E Bonfenoni (Fisher's LSD 


[E Tukey's HSD O Scheffe 
C Sidak C Hochberg's GT2 
Confidence:|0.95 — | 


OKIKI 


You can customize the hypothesis to be tested. You can define contrasts across the 
categories of a grouping factor: 


Hypothesis. Select the type of hypothesis. The following choices are available: 
m Fand R Matrices. Tests the hypotheses corresponding to the F and R Matrices tab. 
m Pairwise. Compare pairs of groups to determine which pairs differ, 


Adjustment method. The following options are available to compute p-value 
adjustments for multiple comparisons: 


W Bonferroni. Uses student's t statistics. It sets the family-wise error rate as 
(1-Confidence)/ (Total number of comparisons). 
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m Fisher's LSD. Equivalent to multiple t tests between all pairs of groups. The 
disadvantage of this test is that no attempt is made to adjust the observed 
significance level for multiple comparisons. 

Sidak. Uses Student's t statistic for pairwise multiple comparisons. 
Tukey's HSD. Uses the Studentized range statistic to make all pairwise 
comparisons. This is the default. 


m Scheffé. The significance level of Scheffé's test is designed to allow all possible 
linear combinations of group means to be tested, not just the pairwise comparisons 
available in this feature. The result is that Scheffé's test is more conservative than 
the other tests. 


m Hochberg's GT2. Uses Studentized maximum modulus distribution. 


Confidence. Specify confidence level for pairwise comparisons tests. The default is 
0.95. 
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F and R Matrices 


To specify F and R matrices in the Hypothesis drop-down list of the 
Mixed Models: Hypothesis Test dialog box: 

The F and R Matrices tab gets enabled. F and R are the matrices of linear weights 
contrasting the coefficient estimates for fixed and random effects respectively. You 
can write your hypothesis in terms of the F and R matrices. 


Analyze: Mixed Models: Hypothesis Test 


| Main | F.F Matices | D Mati 


Fixed effectfs} — 


014 


W Fixed effects. Specify as many numbers as the dimension of your beta vector. In 
case you specify less, SYSTAT takes the unspecified ones as zero; if you specify 
more, SYSTAT ignores the extra ones. 


m Random effects. Specify as many numbers as dimension of your gamma vector. In 
case you specify less, SYSTAT takes the unspecified ones as zero; if you specify 
more, SYSTAT ignores the extra ones. 
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D Matrix 


D is a null hypothesis vector (by default null vector). The D vector, if you use it, must 
have the same number of rows as the F and R matrices. To specify a different D Matrix, 
click the D Matrix tab in the Mixed Models: Hypothesis Test dialog box. 


Analyze: Mixed Models: Hypothesis Test 


Main | FA Matices! D Matix | 


E) Use matrix: 
0.05 


ance of contrasts (rows) in F and R 


Estimate. Check this option for testing signific | 
f the estimable linear parametric 


matrices individually. This test reports estimate o 
function, its standard error and corresponding t-test. 
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Using Commands 


Select the data with USE filename and continue with: 


MIXED 
RESET 
CATEGORY grpvarlist / MISS 
MODEL var = INTERCEPT + varlistl 
RANDOM varlist2 / STRUCTURE - VCOMPONENTS or 
CSYMMETRY 
or UNSTRUCTURED, 
SUBJECT = terml MEANS 
REPEATED / STRUCTURE = VCOMPONENTS or CSYMMETRY 
or AR(1) 
SUBJECT = term2 
SAVE filename / MRESIDUALS CRESIDUALS DATA 
COVPARAMETERS FIXED RANDOM 
SERRORFIXED MODEL 
ESTIMATE / METHOD = REML or ML TYPE = HESSIAN or 
LIKELIHOOD or PARAMETERS 
CRITERION = RELATIVE or ABSOLUTE 
NEM = nl NNR = n2 CONVERGENCE = dl 
HALF = n3 TOLERANCE = d2 


CONFIDENCE = d4 GSTART = [gl, .., gk] 
RSTART = [rl, r2, ., rk] 
To perform hypothesis tests: 
HYPOTHESIS 


PAIRWISE effect / BONF LSD TUKEY SCHEFFE SIDAK GT2 
FMATRIX [matrix] 

RMATRIX [matrix] 

DMATRIX [matrix] 

TEST / CONFI - di ESTIMATE 


Usage Considerations 


Types of data. MIXED requires a rectangular data file. 


Print options. . MIXED displays covariance parameters and tests of fixed effects for 
PLENGTH SHORT. For PLENGTH MEDIUM, MIXED adds fixed effects estimates. For 
PLENGTH LONG, MIXED adds random effects predictions and iteration history. 


Quick Graphs. MIXED produces a quick graph of marginal residuals versus marginal 
predicted values. 
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Saving files. Several sets of output can be saved to a file. The actual contents of the 
saved file depend on the analysis. Files may include estimated regression coefficients, 
model variables, residuals, predicted values, and diagnostic statistics. 


BY groups. Each level of any BY variables yields a separate analysis. 
Case frequencies. MIXED uses the FREQUENCY variable, if present, to duplicate cases. 


Case weights. MIXED uses the values of any WEIGHT variables to weight each case. 


Examples 


Example 1 
Nesting in treatment structure 


Nesting occurs when the values of one categorical variable has different interpretations 
within the values of a second categorical variable. We shall call the first variable the 
NESTED variable, and the second the NESTING variable. In statistical jargon the term 
nested model or hierarchical model is used where the two variables are either both 
treatments or both design variables. If a treatment effect is nested within a design 
effect, it is customary to deal with it as either a split-plot design or a repeated measures 
design. If both the variables correspond to treatments, then we have nesting in 
treatment structure. If both the variables are design variables, then we have nesting in 
design structure. This example will illustrate the analysis of the former case using 
SYSTAT. Nesting in design structure will be dealt with in later examples. 

This example is based on a pesticide data set given in Milliken and Johnson (1992). 
Here we are interested in comparing 11 different brands of pesticides. The first three 
are produced by company 4, the next two by company B, the next two by company C, 
while company D is the manufacturer of the last four brands. To compare these 33 
glass containers are used, which are randomly grouped into eleven groups of three. The 
pesticides are assigned randomly to the groups. The assigned pesticide is applied to the 
inside of each box in its group. Next a box with 400 mosquitoes and soil with bluegrass 


is put inside each container. The number of live mosquitoes in each box is counted after 


4 hours. 
Here we have two treatment effects, company and pesticide. Since the companies 


produce pesticides of different types. the pesticide effect is nested inside company. For 
instance, brand 1 of company A is different from brand 1 of company B. The effects 
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ofthe boxes and containers may be absorbed into the random error of the model, since 
they were assigned at random. A reasonable model to capture this structure is 


yge ^ ua, * Bi + ge 


where k=1,2,3, j=1,....n; =1,...,4, and nj3, n3=2, n3=2, n¿=4. Here y;;y is the 
observation from the k-th box under the the j-th brand from the i-th company. | 


The input is: 


USE PESTICIDE 
MIXED 

CATEGORY COMPANYS PESTICIDE 

MODEL Y = INTERCEPT + COMPANYS + PESTICIDE (COMPANYS) 
ESTIMATE 


Notice that since we are dealing with nesting in treatment structure, we do not have any 
random effect. 


The output is: 

Dependent Variable : Y 

Fixed Factor (s) : COMPANYS, PESTICIDE (COMPANYS) 
Fixed Covariate(s) : Intercept 


Estimation Method : ANOVA Type III 


Notice that that SYSTAT is using ANOVA Type III estimation here, even though we 
have not asked for it. This is because in absence of random effects, this is the default 
estimation method. As we have seen in earlier examples, the default is REML when 
random effects are present. 


Dependent Variable : Y 

Fixed Factor (s) 1 COMPANYS, PESTICIDE (COMPANYS) 
Fixed Covariate(s) : Intercept 

Estimation Method : ANOVA Type III 


Dimensions 


Covariance Parameters : 
Columns in X 
Columns in Z 
No. of Observations 


Error Terms 


COMPANYS 
PESTICIDE (COMPANYS) 
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Analysis of Variance 


Type III SS Numerator df Denominator df 


COMPANYS 22515.477 3 22.000 
PESTICIDE (COMPANYS) 1412.583 7i 22.000 
ERROR ; 1332.000 22 


Analysis of Variance (contd...) 


Source | Mean Squares F-ratio p-value 
COMPANYS i 7505.159 123.959 0.000 
PESTICIDE (COMPANYS) | 201.798 3.333 0.014 
ERROR i 60.545 


Both the p-values are significant (below 0.05, say). We always consider the higher 
order terms first. Here the highest order term is the nested effect, which is significant. 
This means that the different pesticides produced by the same company differ 
significantly among themselves. More specifically, there is at least one company, at 
least two pesticides of which differ significantly. A latter table in the output will shed 
more light on this issue. Since the nested term is found significant, we must be careful 
in our interpretation of the p-value for the main effect due to the companies. Saying 
something like "The typical pesticide of one company differs from the typical 
pesticide from other companies" is not entirely correct, since owing to the significant 


nested effect, there is nothing called a "typical pesticide of a company". 


Estimates of Variance Components 


i Variance 95.00% Confidence Interval 
| Components Standard Error z p-val Lower Upper 
+ 


Estimates of Fixed Effects 


Effect 


pepe E E EE EE O 


Intercept 


PESTICIDE (COMPANY$) n . à y "ede 
1m). 22 . 0.157 876 
ico 22 2.361 0.028 
SIBI d 22 -2.099 0.048 
2(0) | 22 -3.043 0.006 
3(D) } . 22 0.052 0.959 
āo) |, 0.000 Éí 


This table reports the individual t-tests. The rows with dots correspond to the 
coefficients that are assumed to be 0 as part of the estimability constraint enforced by 
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SYSTAT. For each company, the last nested coefficient is assumed to be 0. The row 
3(A) is one such row. This means that pesticide 3 of company A is considered as the 
reference for company A. The other pesticides of the same company will be reported 
with respect to this reference. The insignificant p-value (above 0.05, say) for row 2(A), 
for example, means that the pesticide 2 of company A does not perform significantly 
differently from pesticide 3 of the same company. 


In fact, the pesticides of company A all perform more or less the same. The pesticides 
produced by company B are also similar among themselves. The same, however, 
cannot be said for the other two companies. 


Confidence Intervals of Fixed Effects Estimates 


95.00% Confidence Interval 


Estimate Lower Upper 

79.350 97.983 

27.824 54.176 

38.824 65.176 

717.509 8.842 

PESTICIDE (COMPANYS) . -3.842 22.509 

71.333 714.509 11.842 
0.000 

1.000 712.176 14.176 
0.000 

15.000 1.824 28.176 
0.000 

-13.333 -26.509 -0.158 

-19.333 -32.509 76.158 

0.333 -12.842 13.509 
0.000 


Type III Tests for Fixed Effects 
Source | Numerator df Denominator df F-ratio p-value 
---------------2----- A enema 


COMPANYS 3 22.000 123.959 0.000 
PESTICIDE (COMPANYS) | ri 22.000 3.333 0.014 


Example 2 
Nesting in Design Structure 


In this example (Milliken and Johnson, 1992), we consider an experiment to study the 
effects of temperature on the comfort level of men and women. The experiment was 
carried out using nine environmental chambers, 18 men and 18 women as follows. 

Three different temperatures (65F, 70F and 75F) were assigned to three randomly 
selected chambers. Two randomly selected men and two randomly selected women 
were assigned to each chamber. The comfort of each person was measured after three 
hours in a scale of 1 to 15, where 1=cold, 8=comfortable and 15=hot. 
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Comfort experiment layout 


Here the temperatures are the only treatments, gender and chamber being design 
effects. 


A model for this data set is as follows: 
Yj = H+ + Ya t ja) * Eia 


where yjj, is the comfort measurement for the /-th person of k-th gender inside the j-th 
chamber under temperature i. We shall consider the effects involving chamber as 


random. 
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The input is: 


USE COMFORT 

MIXED 

CATEGORY TEMP CHAMBER GENDER 

MODEL COMFORT = INTERCEPT + TEMP + TEMP*GENDER 
RANDOM CHAMBER (TEMP) 

ESTIMATE 


The output is: 
Dimensions 


Covariance Parameters : 
Columns in X 
Columns in Z 
No. of Observations 


Fit Statistics 


Final L-L : -61.189 
-2L-L : 122.379 
AIC : 126.379 
AlC(Corrected) : 126.823 
BIC : 129.181 


Estimates of Covariance Components 


Description Estimate 


por acsi ccce OI AA e 


CHAMBER (TEMP) Variance 2.358 
Parameter 

Error variance | Variance 1.653 
| Parameter 


Estimates of Fixed Effects 


Effect Level | Estimate Standard Error df t p-value 


TEMP*GENDER 0.742 24 1.572 0.129 
0.000 . A 
0.742 24  -2.695 0.013 
0.000 . a 
0.742 24 -1.347 0.190 
0.000 s 
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We see that temperature and gender make significant contributions to comfort levels, 
since quite a few p-values fall below 0.05, say. 


Confidence Intervals of Fixed Effects Estimates 


i 95.00% Confidence Interval 
Upper 


TEMP*GENDER 


-3.532 -0.468 
-2.532 0.532 
Predictions of Random Effects 
Effect Effect Level | Estimate Standard Error df t p-value 
Eur ca ENT AMPLE Le ES e ce 
CHAMBER (TEMP) — 1(65) i  -0.355 
2(65) 4 1.135 
3(65) i -0.780 
4 (70) H 0.922 
5(70) i -0.780 
6(70) i -0.142 
705) i 2.269 
8(75) i -0.496 
9(75) i -1.773 


Confidence Intervals of Random Effects rs 
95.00% Confidence Interval 


Estimate Upper 


Effect Effect Level 
CHAMBER (TEMP) 1(65) 

2(65) 

3(65) 

4(70) 

5(70) 

6(70) 

705) 

8(75) 

9(75) 


A ORO ET FEM. ou 


Type III Tests for Fixed Effects 


Effect | Numerator df Denominator df F-ratio 
TEMP f » 
TEMP*GENDER | 3 24 3.849 0.022 


None of the design effects are significant (p-values above 0.05, say). 
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Example 3 
Treatment or design? 


Sometimes it may be slightly tricky to determine whether nesting occurs in the 
treatment structure or the design structure. The following example furnishes a case 
where nesting actually occurs in the treatment structure, though it might appear 
otherwise at first. 

This data set is from Bliss (1967). An experiment was conducted to test the 
performance of laboratories and technicians to determine the fat content of dried eggs. 
To this end a single can of dried eggs was stirred well, and 12 samples were drawn. A 
pair of samples (claimed to be of two “types”) was sent to each of six commercial 
laboratories to be analyzed for fat content. Each laboratory assigned two technicians, 
who each analyzed both "types". 

The dependent variable is the fat content measured by each technician for each 
sample. The factors in the design are laboratory, technician, and "type". Here "type" is 
a control effect, while laboratory and technicians are treatment effects (one may treat 
each technician in each laboratory as a measurement method.). 

The experiment has a hierarchical treatment structure. In each of the six 
laboratories, two technicians examined the fat content of the eggs, but the technicians 
in each lab were different, so “Technicians” are nested within “Lab”. Any technician 
effect makes sense only within a single laboratory. For instance, looking for a 
“technician 1" main effect is absurd in this design, because technician 1 of one 
laboratory may not have anything to do with technician 1 of the other laboratory. 

We shall model the data as 


Yi ^ H+ ai + Bic + Yk * £iga 


where Y;jx is the fat content as measured by the j-th technician of the i-th laboratory 
for the k-th type sample sent to the laboratory. Here we know that the types are actually 
fakes. So we shall treat the y, *s as random effects. All the remaining effects are 
considered fixed. 


The input is: 


USE EGGS 

MIXED 

CATEGORY LAB TECHNICIANS SAMPLE 

MODEL FAT = INTERCEPT + LAB + TECHNICIANS (LAB) 
RANDOM SAMPLE 

ESTIMATE 
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The output is 

Dimensions 

Covariance Parameters : 2 
Columns in X :19 
Columns in 2 z 2 
No. of Observations : 48 
Iterations History 


Iterati | Iteration type -2L-L Convergence 


0 -50.607 

1 -50.740 0.003 
2 -50.813 0.001 
3 -50.853 0.001 
4 -50.875 0.000 
5 -50.886 0.000 
6 -50.901 0.001 
7 -50.901 0.000 
8 -50.901 0.000 


Fit Statistics 


Final L-L 1. 25.450 
-2L-L : -50.901 
AIC : -46.901 
AlC(Corrected) : -46.537 
BIC : 743.734 


Estimates of Covariance Components 


Random Effect | Description Estimate 


Variance 0.001 
Pi 


n 
E 
[3l 
m 


Error variance 


| Parameter 


Estimates of Fixed Effects 
p-value 


Effect Level | Estimate 


TECHNICIAN (LAB) 10) | 
201) | 
10) Y 0.050 
2(2) | 0.000 
1(3) ! -0.075 
25. 1 0.000 
1(4) 4 70.002 
2(4) | 0.000 
145) 1 0.012 
2(5) | 0.000 
116) | 0.185 
216) | 0.000 
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As always, we start by considering the higher order terms first. The p-values less than 
0.05 will be considered significant. For laboratories 1 and 6, the two technicians differ 
significantly. For the other laboratories the technicians have come up with more or less 
similar measurements. The average measurements of the laboratories are significantly 
different, as seen by the low p-values for the lab main effect coefficients. 

Predictions of Random Effects 

Effect Effect Level 1 Estimate Standard Error df t  p-value 


SAMPLE 1 i 0.017 0.023 35 0.735 0.467 
2 -0.017 0.023 35 -0.735 0.467 


The measurements of different types do not differ significantly. This is not a mere 
consequence of the fact that the types are actually fakes. A significant difference here 
would signal some foul play in the entire experiment, e.g., if some dishonest, lazy 
technicians have provided false measurements and have put significantly different 
values for the two types just to make the false measurements look "more realistic". 


Type III Tests for Fixed Effects 


F-ratio p-value 


10.215 


TECHNICIAN (LAB) | 4.155 0.001 


This shows something we have already concluded: the measurements from the 
laboratories differ significantly and so do measurements taken by different technicians 
within the same laboratory. 


Example 4 


Nesting 


versus Crossing 


A nested term in a model looks like Y jj while a crossed (interaction) term looks like 
Y ij. Both the subscripts are made of the same two indices i and j. Then what is the 
difference between them? Of course, they have completely different interpretations. 
However, mathematically, they are essentially the same. This example aims to 
elucidate this point. 

This data set, which is from Beckman et al. (1987), has been analyzed in Hocking 
(1985) (p 448). This is a study of high efficiency particulate air (HEPA) cartridges. The 
aim is to compare two types of aerosols used to test the HEPA respirator filters. For 
this two aerosol types were used with 3 filters from each of two different 
manufacturers. Since the filters were unique to the manufacturer, we treat the filter 
effect as nested inside the manufacturer effect. 
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We shall use the model 
Yik = Bij * Oka) t Pika) * Eijkr 


where y;;,, is the r-th observation for the k-th filter from the i-th manufacturer on the j- 
the aerosol. It is assumed that the filters constitute a random sample from a large 
population of filters. So we treat t x's and p g's as random effects. Beckman et 
al.(1987) used a simpler model without the interaction term. 


The input is: 


USE AEROSOL 
MIXED 
CATEGORY MANUFACTURER FILTER AEROSOL 


MODEL Y = MANUFACTURER*AEROSOL 
RANDOM FILTER(MANUFACTURER) * AEROSOL*FILTER (MANUFACTURER) 


ESTIMATE 


Since this example wants to show the similarity between crossed and nested terms 
we shall later run the same SYSTAT program with the nested terms replaced by 
crossed terms. We shall not present the complete output in either case. We shall only 
present the parts that need to be compared. 


The output is: 


Estimates of Covariance Components 
Description Estimate 


Random Effec 


IRER) Variance 0.000 
Parameter 


Standard Error df t 


1*1 (1) 


AE * FA- 3 
ROSOL*FILTER(MANUFA- 327 (P) -0.199 


d 1:30) $ -0:207 0:520 24 -0.397 
1-1 (2) 1 0.553 0.520 24 1.062 
1:20) i -0.256 0.520 24 -0.492 


1*3(2) 
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2*1) i -1.162 0.520 24 =2.233 
2*2(1) E 1.358 0.520 24 2.611 
2*3(1) H -0.197 0.520 24 -0.378 
2°1(2) H -0.191 0.520 24 -0.367 
2*2(2) t -0.361 0.520 24 -0.694 
2*3(2) H 0.552 0.520 24 1.061 

Predictions of Random Effects (contd...) 

Effect 

FILTER (MANUFACTURER) 


AEROSOL* FILTER (MANUFA- 1*1(1) 
CTURER) 1*2 (1) 
1*3(1) 
1*1(2) 
1*2(2) 
1*3(2) 
2*1) 
2*2(1) 
2*30) 
2*1(2) 
2*2(2) 
2*3(2) 


"iem MEE EL uda 


Let us make a mental note of some of the values in this table. We shall later compare 
them with the corresponding values when crossing replaces nesting. The row for 
1*1(1) is for aerosol | used with filter | made by manufacturer 1. The estimate is 0.405. 
The p-value is 0.442. 

Next we replace the nested terms by crossed (interaction) terms. 


The input is: 


USE AEROSOL 

MIXED 

CATEGORY MANUFACTURER FILTER AEROSOL 

MODEL Y = MANUFACTURER*AEROSOL 

RANDOM FILTER*MANUFACTURER 4 AEROSOL*FILTER*MANUFACTURER 
ESTIMATE 
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The output is:: 
Estimates of Covariance Components 
Description Estimate 


Variance 
Parameter 


Variance 0.638 


Variance 0.302 


Error variance i 
i Parameter 


Predictions of Random Effects 
Effect Effect Level 


AEROSOL*FILTER*MANUFA- 1511 
CTURER 1*1*2 


FILTER (MANUFACTURER) 


Predictions of Random Effects (contd.. 
Effect Effect Level 


AEROSOL* FILTER*MANUFA- 
CTURER 


FILTER (MANUFACTURER) 
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We shall now compare this table with the corresponding values from the earlier output. 
The row for 1*1*1 here corresponds to 1*1(1) earlier. The estimate is again 0.405, and 
the p-value is 0.442, as before. 


Example 5 
A Nested-Factorial Model with Case Frequencies 


This example focuses on two aspects of the SYSTAT MIXED command: analyzing a 
nested-factorial model and using case weights. A nested-factorial model is a mixed 
effects model where both crossing and nesting are present. 

Here the data set, which comes from Hocking (1985), is about the concentration of 
phosphorus in the wash water. The aim of the investigation is to determine how the 
concentration varies with the types of detergent and washing machines. The 
experiment was carried out with 4 different types of detergents, 3 different types of 
machines, and 7 laundromats. The laundromats had different numbers of machines, but 
each laundromat had only machines of a single type. Thus, laundromats are nested 
inside machine types. The machines within each laundromat were divided into 4 
groups of roughly equal sizes, and the 4 types of. detergent were allocated to them, The 
response is the average amount of phosphorus in grams per liter from daily one-hour 
samples over a seven day period. The observations have been averaged over all the 
machines assigned to a single detergent type in each laundromat, 

Here the different observations are averages over different numbers of 
measurements, and contain different amounts of information, We shall take this into 
account by considering N as the frequencies of cases. 


We shall try to fit the following nested-factorial model: 
Viger = B+ a; +B; + yi + öka) + Ej) 


where 7=1, 2, k=1,..., ni j71,...,4, 1, 2, 3, and 22, ny-3, n=2. Here yj, is the r-th 
observation when the j-th detergent is used in the k-th laundromat with machines of 
type i. 
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The input is: 


USE PHOSPHOR 
FRQUENCY N 
MIXED 
CATEGORY LAUNDRY DETERG MACHINE 
MODEL Y = INTERCEPT + MACHINE + DETERGENT + MACHINE*DETERG 
RANDOM LAUNDRY (MACHINE) + DETERG*LAUNDRY (MACHINE) 
ESTIMATE 


This input is somewhat different from others used in this chapter because here we are 
using case frquencies. 


The output is: 

Fit Statistics 

Final L-L : -25.145 
-2L-L : 50.290 
AIC + 54.290 
AIC(Corrected) : 55.213 
BIC : 55.835 


Estimates of Covariance Components 


Random Effect | Description Estimate 

EU ae Rape oat uci A cie 

LAUNDRY (MACHINE) | Variance 0.041 
; Parameter 

poa rm Pollucis EE 

Error variance | Variance 0.471 
| Parameter 

Estimates of Fixed Effects 


Estimate 


Effect Level 


MACHINE*DETERG 


A A A Mos card 
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Predictions of Random Effects 


LAUNDRY (MACHINE) 


2(1) 0.030 
1(2) 70.173 
2(2) 0.037 
3(2) 0.136 
1(3) -0.054 
2(3) 0.054 


Type III Tests for Fixed Effects 


Effect H nominator df F-ratio p-value 
Lx. a LUE H a Lali aia, E- aiaa 
MACHINE H 4 13.670 0.016 
DETERG i 12 19.654 0.000 
MACHINE*DETERG | 12 4.044 0.019 
Example 6 
Confidence Intervals 


The SYSTAT MIXED command can compute confidence intervals for user specified 

contrasts in various inference spaces (broad, intermediate, and narrow.) In this example 
we shall explore this with a data set from Brownlee (1960) (Hocking, 1985, p 535). The 
experiment seeks to compare two different annealing methods for making cans. Three 
coils of material were selected from the populations of coils made by each of the two 


methods. A pair of samples was drawn from each of two locations on the coil. The 
response is the life of the can. 


Following Hocking, we shall fit the model 
Yijk 7 Hij * Aka) + Bika) + Eijk 


where i is for method, k for coil within method, and j is for location. Here ot ki and 


P iko) are random effects. Our aim is to produce a 90% confidence interval for the 
difference between the two methods, i.e., for the contrast. 


Hit + Hio — Hoi — Ho; 
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We shall use the broad inference space (the default.) First we need to fit the model 
using the SYSTAT input: 


USE ANNEEAL 
MIXED 
CATEGORY METHOD LOCATION COIL 
MODEL LIFE = METHOD*LOCATION 
RANDOM COIL(METHOD) LOCATION*COIL (METHOD) 


ESTIMATE 
The output is: 
Fit Statistics 
Final L-L -83.771 
-2L-L 167.542 
AIC 173.542 
AlC(Corrected) : 175.042 
BIC : 176.530 


Estimates of Covariance Components 
e | Description Estimate 
* 


Rand Eff 


COIL (METHOD) 


Error variance 


| Parameter 


Estimates of Fixed Effects 
1 | Estimate Standard Error df t p-value 


Effect | Level | Estimate Standard Error df == f | p-value 
METHOD LOCATI $ 329.833 | 14.485 4 22.770 0.000 
idt 12 1 14.485 4 21.436 0.000 
2*1 1 14.485 4 21.205 0.000 
2*2 i 291.167 14.485 4 20.101 0.000 


Predictions of Random Effects 
Effect Level | Estimate Standard Error df t p-value 


Eff 


COIL (METHOD) 


LOCATION*COIL (METHOD) 
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1*3(1) i 0.000 
1*1 (2) i -0.001 
1*2 (2) i 0.000 
1*3 (2) i 0.001 
2*1 (1) i 0.000 
2*2(1) i 0.000 
2*3(1) i 0.000 
2*1 (2) i 0.001 
2*2 (2) i 0.000 
2*3(2) i -0.001 


Plot of residuals against predicted values 


RESIDUAL 


-50 
290 300 310 320 330 
ESTIMATE 


Type III Tests for Fixed Effects 


Effect df Denominator df F- 
METHOD* LOCATION 
F Matrix 

1 2 3 4 


1.000 1.000 -1.000 -1.000 


F-ratio Test 


Numerator df | Denominator df F-ratio p-value 


0.000 
-0.009 
-0.003 
.012 


coooom-ooorc- 


.000 
.993 
.998 
.991 
.000 
+999 
.999 
.994 
.997 
.991 
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Example 7 
Nested Random Effects 


This example is based on a dataset given by Robinson (1987) (Kuehl, 2000). Two mass 
spectrometers (SPECTROMTRS) were compared for accuracy in measuring the ratio 
of N to ^N. Three plots of land (PLOT) treated with SN were used and from every 
plot two soil samples (SAMPLE) were taken. Each sample had two observations. The 
response variable RATIO is the ratio of MN to N multiplied by 1000. 

Here PLOT is a random effect and SAMPLE is another random effect nested in 
PLOT. MACHINES is a fixed effect. That is, the SAMPLE within PLOT is a subject 
on which repeated observations are taken. 

We can perform the mixed models in two ways: Take PLOT as a random effect and 
SAMPLE as another random effect with grouping factor PLOT, or take PLOT as a 
random effect and specify compound symmetry structure for errors and take 
PLOT*SAMPLE interaction as a grouping factor of errors. Both the approaches 
essentially have the same interpretations. We will use the latter approach. 


The input is: 


USE SPECTROMETERS 

MIXED 

CATEGORY PLOT SAMPLE 

MODEL RATIO - INTERCEPT * SPECTROMTRS 


RANDOM PLOT 
REPEATED / STRUCTURE = CS GROUP - PLOT*SAMPLE 


ESTIMATE 


The output is: 
Dimensions 


riance Parameters : 2 
mns in X 1:32 
2 3 

4 


lumns in Z t 
No. of Observations :2 


Fit Statistics 
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Estimates of Covariance Components 
Random Effect Description Estimate 


See pp nuire a ia RECTUS 


Error variance | Variance 0.002 
Parameter 

Error 0.905 
Correlation 

(cs) 


Our subject here is PLOT*SAMPLE. Value 0.905 indicates the higher significance of 
within-subject correlation. 


Estimates of Fixed Effects 
Effect Level | Estimate Standard Error df t p-value 
Intercept 


----------------2---24-. 
SPECTROMTR$ A | | -0.065 9:005. 20 -10.543 0.000 


Effect Effect Level | Estimate Standard Error df t p-value 


1 
i 

a ——ü A" = 
' 


PLOT 1 0.030 0.034 20 0.894 0.382 
2 0.009 0.034 20 0.278 0.784 
3 -0.040 0.034 20  -1.172 0.255 


Type III Tests for Fixed Effects 
Effect Dei 


SPECTROMTR$ | 1 20 111.151 0.000 


The test of fixed effects indicates that the readings of two spectrometers are 
significantly different. 
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Mixed Regression 


Donald Hedeker, Rick Marcantonio, and Michael Pechnyo 


Mixed regression estimates models containing combinations of fixed and random 
effects for response data having a normal distribution. Mixed models, or multilevel 
models, have also been referred to as “hierarchical linear models" (Bryk and 
Raudenbush, 2001), "random coefficient models" (deLeeuw and Kreft, 1986), and 
"variance component models" (Longford, 1993). The implementation here 
corresponds to the MIXREG program of Hedeker and Gibbons (1996). 

These models require a data structure in which observations having a common 
characteristic can be classified into identifiable groups, known as level-2 units, 
resulting in nesting of the observations within the level-2 units. Mixed regression uses 
random effects to account for dependencies in the data due to this nesting structure, 
allowing simultaneous analysis of individuals and the groups to which the individuals 
belong. For an individual level-2 unit i, the model for mixed regression is: 


y, = Wia + Xi + £i 


le, W is a design matrix for fixed effects, o. is a vector 
X is a design matrix for random effects, B is a vector 
e is a vector of residuals. Models without random 

n models, but use marginal maximum likelihood to 


where y is the dependent variab 
of fixed regression parameters, 
of effects specific to unit i, and 
effects parallel standard regressio 
derive the parameter estimates instead of least-squares techniques. 


Researchers often use mixed regression for the analysis of both clustered and 


longitudinal data. In clustered data, observations from different subjects are nested 
within a larger group, such as students within schools; random effects represent 
differences between the clusters. In contrast, for longitudinal data, observations are 
nested within each subject. In this case, the individual can be viewed as the "cluster", 


so random effects represent differences between subjects. 
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Mixed regression, ANOVA, and general linear models can all be used for repeated 
measures analysis. However, unlike the other two procedures, mixed regression 
analyzes unbalanced data. Additionally, you can include an autocorrelation structure 
to model the relationships in the residuals over time. 

For each model you fit, the software reports parameter estimates, correlations 
between estimates, and the intraclass correlation coefficient. You can also view 
empirical Bayes estimates of the parameters for the random effects. A variety of 
statistics, including level-1 and level-2 residuals and predicted values can be saved to 
a file for further analyses and plotting. 


Statistical Background 


Mixed regression is a modeling technique designed for the analysis of multilevel data. 
In multilevel data, individual, or level 1, observations can be classified as belonging to 
known groups, or level 2 units. The data are nested within these groups, leading to a 

hierarchical structure. The number of observations can vary across level 2 units. The 
standard data structure appears below: 


“Clustered” is one common type of multilevel data. In this situation, we have 
observations from different subjects who can be classified into groups. For example, 
we may have measurements from students from different classes or schools. 
Alternatively, we can consider patients nested within doctors, clinics, or hospitals. The 
goal of the analysis is to examine the effects of variables at both the individual and the 
group levels. 
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A second type of multilevel data occurs when collecting repeated measures on each 
subject. In this case, the level 2 unit corresponds to the person; observations are nested 
within individuals. The researcher collects measurements over time to examine the 
effects of time-invariant and time varying variables. 

In introducing the basic notions behind mixed regression, we will use a subset of 
clustered data from the Junior School Project. In the data we examine, roughly 1400 
students from 49 schools provided a Ravens test score of ability and a score in 
mathematics. Because we are using the data for illustrative purposes only, we will not 
discuss other variables measured, but refer the reader to Mortimore et al. (1988) or the 
Multilevel Models Project home page (www.ioe.ac.uk/multilevel/) for the complete 


data file. 
Historical Approaches 


In the past, several different techniques have been applied to multilevel data. Consider 
the following two plots: 


50 
40 


30 


0 10 20 30 40 
Ravens Test 


Ravens Test 


In the first plot, the regression line ignores any effect due to different schools. 
Essentially, we are treating all of the data as if it came from one school. Obviously, we 
portant information and are violating the 


are ignoring some potentially im; 
independence assumption inherent in regression. In the second plot, we fit a separate 


regression line to each school. Interpretation is exceedingly cumbersome and any 


generalization across schools cannot be made. 
Another possibility is to aggregate the level 1 data to the level 2 unit, and perform 


the analysis at level 2. For the current data, this would involve computing mean scores 
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for each school, and using the 49 means in the regression. However, the resulting 
model cannot be applied to individuals and the variation in scores due to individuals is 
lost. Without that information, relationships appear stronger than they otherwise 
would. 


The General Mixed Regression Model 


For level-2 unit i, the general form of the mixed regression model is 
y, = Wo * XB, e; 


where y represents the response vector, W is a design matrix for fixed effects, 0L is a 
vector of fixed regression parameters, X is a design matrix for the random effects, B 
corresponds to a vector of individual effects, and £ is a residual vector. Terms having 
a subscript vary across level-2 units. 

The random effects have a multivariate normal distribution with mean u and 
covariance matrix X. The residuals have an independent multivariate normal 
distribution with mean 0 and covariance matrix 6,7. For independent residuals, Q 
corresponds to the identity matrix, but in general we will allow a variety of 
autocorrelation matrices to model dependencies. 


Autocorrelation 


Five autocorrelation structures can be used in the covariance matrix for the residuals. 


m First-order autoregressive. This model assumes exponentially decreasing 
autocorrelations as timepoints get farther apart. The general form is: 


= Nonstationary first-order autoregressive. This structure allows for exponentially 
decreasing autocorrelations with nonconstant variance over time. 
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m First-order moving average. If O equals the moving average parameter, the general 
form of the autocorrelation matrix is: 


a 
m siis Oa 
1+0 

0 0 
z = 0 0 
1+0 1+0 
0 0 
o --—— S 0 
1+0? 1+0? 
0 
0 E —Á 
120 1+0° 
o o 
140 
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m Autoregressive, moving average (1,1). A combination of a first-order 


autoregressive process having parameter ( with a first order moving average 


process having parameter 0. The general form equals: 


1 o otop op 
o 1 o od od 
op o 1 o ob 
op opo 1 o 
ov od op o 1 


where 
o - (1-999 8) 
1-206 0 


m Toeplitz. General structure having constant autocorrelations along the 
subdiagonals. The structure equals: 


1 pi P2 P3 Pa 
Pi 1 Pi P2 Ps 
P2 Pi 1 Pi Po 
P3 P2 Pi | Pi 
Pa P3 P2 Pi 1 


Fixed Intercept, Fixed Slope Model 


The model containing only fixed effects ignores effects due to the nesting of the 
observations. This model is analogous to the standard linear regression model, but is 


estimated using marginal maximum likelihood instead of least-squares. 
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For the JSP data, the estimated parameters and log-likelihood are: 


Log Likelihood > -3679.1187 


Standardized 
Variable Estimate Error VA p-value 
INTERCEPT 7.7604 0.7598 10.2132 0.0000 
RAVENS TEST 0.6911 0.0295 23.3884 0.0000 
34.4122 1.4326 24.0208 0.0000 


Random Intercept, Fixed Slope Model 


This model uses the intercept to account for level-2 differences. Each level-2 unit 
yields a separate intercept. The slope, however, is a constant across all level-2 units. 
For our data, the following parameter estimates result: 


Log Likelihood = -3659.1380 


Standardized 
Variable Estimate Error 2 p-value 
INTERCEPT 7.4563 0.7925 9.4091 0.0000 
RAVENS TEST 0.6988 0.0296 23.5898 0.0000 
Residual variance: 
31.9436 1.3581 23.5214 0.0000 


Estimate 


INTERCEPT 
1 INTERCEPT 2.2437 
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A plot of this model appears below. 
Random Intercept, Fixed Slope Fixed Intercept, Random Slope 


40 40 


30 30 
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Ravens Test 


Fixed Intercept, Random Slope Model 


This model is used less frequently than the others. In this situation, the slope varies 
across schools, but the intercept is common. The results for the JSP data are: 


Log Likelihood = -3659.3330 


Standardized 
Variable Estimate Error z p-value 
RAVENS_TEST 0.6909 0.0307 22.4726 0.0000 
INTERCEPT 7.6654 0.7567 10.1304 0.0000 
Residual variance: 
31.9987 1.3602 23.5250 0.0000 


Random-effect variance & covariance term(s): 


Estimate 


1 
RAVENS TEST 
1 RAVENS TEST 0.0032 


A plot of this model appears above. 


11-429 


Mixed Regression 


Random Intercept, Random Slope 


The completely random model is the most general. In this situation, both the intercept 
and the slope vary from level-2 unit to level-2 unit. The results for the JSP data are: 


Log Likelihood - -3653.1624 

Standardized 
Variable Estimate Error z p-value 
INTERCEPT 7.1830 1.0642 6.7494 0.0000 
RAVENS_TEST 0.7087 0.0404 17.5327 0.0000 


Residual variance: 


30.8264 1.3361 23.0712 0.0000 


iance term(s): 


Estimate 
1 2 
INTERCEPT RAVENS_TEST 
pl INTERCEPT 24.3553 
2 RAVENS TEST -0.8564 0.0332 


The random effects have a multivariate normal distribution. This distribution appears 
below. 


follows. Compare this plot to the separate 


A plot of completely random model } p 
historical approach. The lines for this model are 


regressions plot used to illustrate an 
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The completely random model is similar to computing separate regressions for each 
level-2 unit. However, mixed regression controls for the group effects in a single 
model. The following plot compares the least-squares estimates for separate 
regressions to the mixed regression estimates. 
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The spread of the intercepts is much less for mixed regression. Similarly, the slopes are 
less variable. Notice though that both sets of intercepts and both sets of slopes are 
centered at the same value. 


Model Comparisons 


To determine whether an effect should be treated as random or fixed, use a likelihood 
ratio test to compare models that treat the effect each way. The statistic [-2*(the 
difference between the log-likelihoods)] has a chi-square distribution with degrees of 
freedom equal to the difference in the number of parameters estimated. 

For example, comparing the completely fixed model to the random intercept, 
random slope model yields a statistic of: 


-2*[-3679.1187 - (-3659.1380)] = 39.96 


The random intercept model adds one parameter to the fixed intercept model, so the 
degrees of freedom for the test equal 1. The p-value for the test is less than 0.0001, 
indicating that the variability of the random intercept is significant. A single fixed 
intercept is inadequate for the JSP data. Similar comparisons can be explored for the 
other models. 


Mixed Regression in SYSTAT 


Mixed Regression: Hierarchical Data 


To open the Mixed Regression: Hierarchical Data dialog box, from the menus choose: 


Analyze 
Regression 


Mixed 
Hierarchical Data... 
or 


Analyze 
Mixed Models 
Mixed Regression 
Hierarchical Data... 
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Regression: Mixed: Hierarchical Data 


| Model i Autocorrelation | Category | Options 

Available variable(s]- =e Dependent — 
{SCHOOL 

| CLASS Lt WP 
| POST. THKS | Fixed effectis): x) 
cc POST. THKS 

TV 

CC TV 
CONSTANTI 


(abusif ce 


[ «Required, 


Random elfect(s]: 


Eis 

© Random 

O Fixed 

Dive mew[ — 4 


EJE 


The following options are available: 


Dependent. Specify a continuous, numeric variable to be predicted from the fixed and 
random effects. 


Fixed effects. Select one or more continuous or categorical (grouping) variables. 
Effects corresponding to the selected variables do not vary across groups. If you want 
interactions in your model, you need to build these components using the Cross button. 
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Identifier. Models containing random effects require an identifier variable to denote 
the nesting structure. Specify a numeric or string variable that identifies group 
membership. For cross-sectional data, this variable corresponds to the cluster ID. For 
longitudinal data, the variable corresponds to the subject ID. 


Random effects. Select one or more continuous or categorical (grouping) variables. 
Effects corresponding to the selected variables vary across groups. If you want 
interactions in your model, you need to build these components using the Cross button. 
An effect specified as random is fit as a random effect and as a fixed effect. As a result, 
you cannot fit models in which there are effects that are random but not fixed. 


Intercept. Mixed regression models can contain an overall intercept, an intercept that 
varies across groups, or no intercept. 


m Random. Include a separate intercept for each group defined by the identifier 
variable. Inclusion of a random intercept corresponds to fitting an effect due to the 


identifier. 
m Fixed. Include an intercept that is constant across groups. 
m None. Omit the intercept from the model. 


Save. You can save three sets of output to a data file: 


m Data. Saves the dependent variable and the design matrix used as mixed regression 
input. Categorical variables result in effect or dummy coded variables in the saved 


file. 
m Bayes. Yields the empirical Bayes estimates for the random effects. 
Residuals. Saves the level-1 and level-2 predicted values and residuals. 


Autocorrelation 


By default, mixed regression assumes the errors are uncorrelated. For longitudinal 
leading to models that include an 


data, this assumption may be unrealistic, ; l , 
autocorrelation structure for error to account for dependencies over time. To specify an 
tab in the Mixed : Hierarchical Data 


autocorrelation structure, click Autocorrelation 
dialog box. 
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Regression: Mixed: Hierarchical Data 


| Model Autocorrelation | Category | Options! f 


Available variable(s! — 
POST_THKS — | 


ram 


Type 

s) Stahonary AR! 
Norrstationary AR(T 
Stationary MA(1] 


Stationary ARMA[T. 1] 


General (T oeplitz] structure 
Number, |1 j ] 


Fix autocorrelation term: 


Selected variable. Specify a numeric variable that represents measurement occasions, 
or "time". Typically, this variable is measured in minutes, hours, or days. 


The following options are available: 


Type. Select one of the following autocorrelation structures for the residual covariance 

matrix: 

m Stationary AR(1). Exponentially decreasing correlations as measurement 
occasions get farther apart. 


11-435 


Mixed Regression 


m Non-stationary AR(1). First-order autoregressive process with nonconstant 
variance over time. 

m Stationary MA(1). Constant, nonzero correlation between consecutive 
observations only. 

m Stationary ARMA(I,1). Structure that is part first-order autoregressive and part 
first-order moving average. 

m General (Toeplitz) structure. Constant, nonzero correlation for a specified lag. 
Enter a lag greater than 0, but less than the maximum number of measurement 
occasions. For each lag smaller than the specified lag, the correlation is also 
constant and nonzero, but need not equal the correlation for other lags. 


Fix autocorrelation terms. Uses specified values for the autocorrelation terms instead 

of estimating them, freeing degrees of freedom. The number of terms to enter depends 

on the autocorrelation structure: 

m Foran AR(1) structure, enter a single value corresponding to the autoregressive 
parameter. 

= For the moving average (MA) structure, enter a single value corresponding to the 
moving average parameter 

= For the ARMA(I,1) process, enter two values, the autoregressive parameter and 
the moving average parameter, separated by a comma. 

= For the Toeplitz structure, the number of fixed terms equals the specified lag. Enter 
the autocorrelations in order from the smallest lag to the largest. Separate the 
values with commas. 


Categorical Variables 


You can specify numeric or character-valued categorical (grouping) variables that 
define cells. You want to categorize an independent variable when it has several 
categories such as education levels, which could be divided into the following 
categories: less than high school, some high school, finished high school, some 
college, finished bachelor’s degree, finished master’s degree, and finished doctorate. 
On the other hand, a variable such as age in years would not be categorical unless age 


were broken up into categories such as under 21, 21-65, and over 65. , 
To define categorical variables, click Category tab in the Mixed : Hierarchical Data 


dialog box. 
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Regression: Mixed: Hierarchical Data 


| Model) Autocorrelation) Category | op 


Available variable(s): - Categorical variable(s]: 


. POST THKS | POST. THKS — 


(Missing values 


Coding 
© Dummy 
O Effect 


The following options are available: 


Missing values. Includes a separate category for cases with a missing value for the 
selected variable(s). 


Coding. You can elect to use one of two different coding methods: 


= Dummy. Produces dummy codes for the design variables instead of effect codes. 
Coding of dummy variables is the classic analysis of variance parameterization, in 
which the sum of effects estimated for a classifying variable is 0. If your categorical 
variable has k categories, k-1 dummy variables are created. 
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m Effect. Produces parameter estimates that are differences from group means. 


Mixed Regression Options 


To specify options for mixed regression models, click Options tab in the Mixed dialog 
box. 


Regression: Mixed: Hierarc hical Data 


Model | Autocorrelation Category Options | 


Number of level 2 units to list: (a y Ui 


Number of EM iterations: [o iy ] 


Convergence [0.0001 ] 


| [V] Reparameterize variance terms 
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The following options are available: 


Number of level 2 units to list. Includes the data for the specified number of groups in 
the output. When using multivariate data, listing the data along with the output can 
confirm the conversion to a hierarchical structure. 


Number of EM iterations. Specify the number of EM iterations to perform before 
switching to Fisher scoring. An EM iteration takes much less time to complete than a 
Fisher scoring iteration, but EM requires far more iterations to arrive at a final solution. 
In an effort to derive final estimates quickly, estimation uses the EM algorithm to 
approach the solution before switching to Fisher scoring to minimize the steps 
required. For models that include autocorrelation terms, set the number of EM 
iterations to 0. 


Convergence. The estimation process ceases and reports a final solution when the 
changes in all parameter estimates between two consecutive iterations are smaller than 
the specified convergence criterion. 


Reparameterize variance terms. Estimation of model parameters may fail if the 
estimated variances for the random effects approach zero. To alleviate this problem, 
the variances can be reparameterized using the exponential transformation. After 
reaching convergence, the estimated variances are returned to their original units via a 
log transformation. 


Data Structure 


Mixed regression analyzes data having a hierarchical structure with an identifier 
variable indicating the nesting of measurements within level-2 units. For example, in 
the following layout, SUBJECT indicates that the first three responses correspond to 
subject 1: 


SUBJECT TIME RESPONSE 
1 ri 
r2 
r3 
nı 


122 


NNN =- 
ay tu wr 


23 
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An alternative structure, often used for repeated measures data, uses multiple variables 
to record the responses within each level-2 unit: 


SUBJECT TIME1 TIME2  TIME3 
1 TH ro t13 
2 ni 12 13 


To analyze data of this type, the software must temporarily reorganize the data into 
a hierarchical layout. 


To analyze data having a multivariate structure, from the menus choose: 


Analyze 
Regression 
Mixed 
Multivariate Data... 
or 


Analyze 
Mixed Models 
Mixed Regression 
Multivariate Data... 
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Regression: Mixed: Multivariate Data 


gs eee 


| Specify a new variable name and variables to wrap under it: 


* [ VEMM 
New variable name: ES 


Available variable(s) 

| SCHOOL 
CLASS 
POST_THKS E 


t- Remove 


In the Data Structure dialog box, select the variables to be stacked. Non-selected 
variables become constants across each associated set of observations in the new data 
set. 


New variable name. Enter a name for the variable under which the selected variables 
should be stacked. This variable typically corresponds to the dependent variable in the 
mixed regression model. 

The restructured data includes two other new variables, CASE and TRIAL. CASE 
corresponds to the case number from the multivariate data. TRIAL reflects the order of 
the selected variables. 

After restructuring the data, define the mixed regression model. Usually CASE 
denotes the nesting of the observations and should be used as the identifier variable in 
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your model. For longitudinal data, TRIAL represents time and can be used as either a 
fixed or random effect. In addition, this variable can form the basis of an 
autocorrelation structure for errors. 


Using Commands 


First, specify your data with USE filename. Continue with: 


MIX 
RESET 
CONVERT newname = varlist 
MODEL depvar = INTERCEPT + fixedvarlist 
RANDOM INTERCEPT + randomvarlist 
IDENTIFIER var 
AUTO var / TYPE=AR or NAR or MA or ARMA or GEN, 
NUMBER=n FIX=valuelist 
CATEGORY varlist / EFFECT or DUMMY, 
Is 


MISS 
SAVE filename / BAYES RESID DATA 
ESTIMATE / NREC-r NEM-m CONV=crit, 
REPAR-ON or OFF 


Usage Considerations 


Types of data. Mixed regression requires a rectangular data file. 

Print options. If PLENGTH SHORT, output includes descriptive statistics, parameter 
estimates, correlations between estimates, and the intraclass correlation coefficient. 
The MEDIUM length adds empirical Bayes estimates to the output. LONG adds the 
iteration history plus variances and covariances for the empirical Bayes estimates. Use 
PLENGTH NONE to suppress all text output. 

Quick Graphs. For models containing one random effect, the Quick Graph displays the 
distribution of the empirical Bayes estimates. Models containing two or more random 
effects yield a scatterplot matrix of the empirical Bayes estimates. 

Saving files. You can save empirical Bayes estimates, residuals with predicted values, 
or the design matrix. Saved files include effect or dummy coded variables in place of 
the corresponding categorical variables 


BY groups. Mixed regression analyzes data by groups. Your file need not be sorted on 
the BY variable(s). However, saved files only include results for the first BY group. 


Case frequencies. The calculations ignore any FREQUENCY variable specifications. 
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Case weights. Weighting of cases is not available in mixed regression. 


Examples 


Example 1 
Clustered Data in Mixed Regression 


To illustrate the use of mixed regression for clustered data, we use data from the 
Television School and Family Smoking Prevention and Cessation Project. Hedeker 
and Gibbons (1996) looked at the effects of two factors on tobacco use for students in 
28 Los Angeles schools. One factor involved the use ofa social-resistance curriculum 
or not. The other factor was the presence or absence of a television intervention. 
Crossing these two factors yields four experimental conditions, which were randomly 
assigned to the schools. Students were measured on tobacco and health knowledge 
both before and after the introduction of the two factors. 

First, we ignore the effects of the nesting within classes by applying a model that 
includes fixed effects only. 


The input is: 


MIX 

USE TVFSP 

RESET 

MODEL POST_THKS = INTERCEPT+PRE_THKS+CC+CC*TV+TV 
SAVE RESIDUALS1 / RESID d 

PLENGTH SHORT 

ESTIMATE 


USE RESIDUALS1 

LABEL CC / 0='No', 1='Yes' 

LABEL TV / 0='No', 1='Yes' 

PLOT PREDO*PRE_THKS / OVERLAY GROUP=CC TV SMOOTH=LINEAR, 
SHORT DASH=9,1,4,10 SIZE= 0, 
YLAB='Post-intervention THKS', 
XLAB='Pre-intervention THKS' 


We save the residuals and predicted values to view the results of the model graphically. 
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Mixed Regression 


Terms in the analysis and names of design matrix columns used for those terms: 


ce Y TV 

FXD4 

Perform 10 EM iterations 
0 random terms 

5 fixed terms 


Numbers of Observations 


Level 2 observations : 1600 
Level 1 observations : 1600 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 
POST THKS 0.000 .000 2,662 1.383 
INTERCEPT 1.000 1.000 1.000 0.000 
PRE THKS 0.000 6.000 2.069 1.260 
cc 0.000 1.000 0.477 0.500 
TV 0.000 1.000 0.499 0.500 
FXD4 0.000 1.000 0.239 0.427 


Starting Values 


Covariates: 
1.661 0.325 0.641 0.199 -0.322 
Residual: 


1.693 
Final Results - MML Estimates 


EM Iterations : 10 
Fisher Iterations : 2 
Total Iterations +: 12 
Log Likelihood : -2688.962 


standard Error 


Variable Esti: 


INTERCEPT 1.661 0.084 
PRE_THKS 0.325 0.026 
cc | 0.641 0.092 
TV 0.199 0.090 
FXD4 -0.322 0.130 


Residual Variance 


Standard Error 


28.284 0.000 


Estimate 


Correlation of the MML Estimates of the Fixed Terms 


1 INTERCEPT 
2 EREMO 1.000 
jew 01486 1.000 

5 0-10) 0.690: -1.000 


FXD4 
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a 


e 
xa 
P 
E 
c 
2 
$3 
z 
2 
$), CC,IV 
32 - No, No 
a E No, Yes 
Yes, No 
~~ Yes, Yes 


obe. pr 


p a Nema. 
Pre-intervention THKS 


The output begins with a note about naming conventions used. The CC*7V interaction 
receives a root name of FXD because the interaction is a fixed effect. The digit 
appended to this root corresponds to the position of the effect in the model 
(INTERCEPT = 1; PRE THKS -2; CC = 3; CC*TV = 4; TV = 5). All subsequent 
references to FXD4 represent the interaction between CC and TK. 

Looking at the p-values for the effects, we find a significant effect for all variables 
in the model. Due to the cross-classification of the CC and TV variables, we actually 
fit four parallel regression lines. 


CC TV Regression Line 
0 0 POST THKS = 1.6613 + 0.3252*PRE THKS 
0 1 POST_THKS = 1.8600 + 0.3252*PRE_THKS 
1 0 POST_THKS = 2.3019 + 0.3252*PRE_THKS 
1 1 POST_THKS = 2.1790 + 0.3252*PRE_THKS 


These lines correspond to those shown in the plot of the predicted values, 
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Random Intercept Model 


In the TVFSP data, the students can be treated as nested within classes, or as nested 
within schools. In this example, we consider the effects of the nesting within classes. 
To account for the data clustering, we use a random intercept. 


The input is: 


MIX 

USE TVFSP 

RESET 

MODEL POST THKS = PRE_THKS+CC+CC*TV+TV 
IDENTIFIER CLASS 

RANDOM INTERCEPT 

SAVE RESIDUALS1 / RESID 

PLENGTH SHORT 

ESTIMATE 


In contrast to the model containing fixed effects only, the random intercept model fits 
four regression lines for each school. Because we treat PRE THKS as a fixed effect, 


the regression lines are all parallel. 


The output is: 


Terms in the analysis and names of design matrix columns used for those terms: 
CC * TV 

FXD3 

Perform 10 EM iterations 

1 random terms 

4 fixed terms 


Numbers of Observations 


Level 2 observations : 135 
Level 1 observations : 1600 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 
POST THKS 0.000 7.000 2.662 1.383 
INTERCEPT 1.000 1.000 1.000 0.000 
PRE THKS 0.000 6.000 2.069 1.260 
cc” 0.000 1.000 0.477 0.500 
TV 0.000 1.000 0.499 0.500 
FXD3 0.000 1.000 0.239 0.427 
Starting Values 
Mean: 

1.661 

Covariates: 


0.325 0.641 0.199 -0.322 
Variance Terms: 

0.339 
Residual: 

1.693 
Final Results - MML Estimates 
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EM Iterations 10 
Fisher Iterations 7 
Total Iterations 17 
Log Likelihood + -2679.982 
Variable Estimate p-value 


Residual Variance 
Esti 


te Standard Error 2 p-value 
1.603 0.059 27.200 0.000 
Random-Effect Variance & Covariance Term(s) 


Estimate 


1 
INTERCEPT 


Pet 


Standard Error 


z 
H 1 
| INTERCEPT 
3.146 
t 1 
| INTERCEPT 
1 INTERCEPT | 0.001 


Note: p-values are 2-tailed except for those associated with variances, 
which are l-tailed. 


Calculation of the Intracluster Correlation 


Residual Variance : 1.603 

Cluster Variance : 0.087 

Intracluster Correlation : 0.087 / (0.087 + 1.603) = 0.051 
Correlation of the MML Estimates of the Fixed Terms 


1 2 5 
INTERCEPT PRE THKS cc TV FXD3 


1 INTERCEPT 
2 PRE_THKS 
d te 
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4 TV i -0.593 0.019 0.486 
4 i . . . 1.000 
5 FXD3 i 0.408 -0.005 -0.707 -0.695 1.000 


Correlation of the MML Estimates of Variance-Related Terms 
2 


— Residual 
1 verceei Ti rona 
2 Residual | 1.000 
Empirical Bayes Estimates 
AAA A | 


8 
T 
€ 
8 
d uoodold 


5 
aaf 
o 
o 
S 
("9 
sb 2 
ol 0.0 
1.0 20 25 


15 
INTERCEPT 


The output includes the number of observations at each level for the mixed model. The 


number of level-2 observations corresponds to the number of groups in the analysis. In 


this case, the students are nested within 135 classes. The number of level-1 


observations indicates the total number of students, 1600. Use PLENGTH MEDIUM to 


view the number of students within each class. 
The individual tests for the parameter estimates indicate significance of the pre- 


intervention score and of the social-resistance curriculum. In contrast to the fixed 


effects only model, however, the television intervention and the interaction do not 


exhibit significant results. Accounting for the classroom effect leads to different 
conclusions than when we ignore clustering. 
The Quick Graph displays the distribution of the empirical Bayes estimates of the 


intercepts. These values appear to be normally distributed about a value of 1.7. Plotting 


the predicted values for this model helps to illustrate the effect of fitting a random 


intercept. 
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The input is: 


USE RESIDUALS1 

LABEL CC / 0='No', 1='Yes' 

LABEL TV / 0-'No', 1-'Yes' 

BEGIN 

PLOT PRED1*PRE THKS / MULTIPLOT GROUP=CC TV, 
YLAB-'Post-intervention THKS', 
XLAB-'Pre-intervention THKS', 
FTITLE-OFF 

PLOT PREDO*PRE THKS / MULTIPLOT GROUP-CC TV, 
SMOOTH-LINEAR SHORT SIZE=0, 
COLOR-RED YLAB-'' XLAB-'' 

END 


The output is: 


w = 


N 


SMHL uonusAiejur]sod 


SHHL uonueAJeju-]sog 
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The line indicates the average trend for each CC, TV combination. The points 
correspond to the individual trajectory for each class. The fixed effect model generates 
a single predicted value for each PRE_THKS value. The random intercept model 
generates a predicted value for each PRE_THKS value within each class. When we 
allow each class to have a regression line, we eliminate the TV and interaction effect 
present when all classes employed a common regression line for each CC*TV 
combination. 


Example 2 
Categorical Variables and Clustered Data 


This example uses a subset of data from the Inner London Education Authority (ILEA) 
analyzed by Goldstein, H.(1987). For 2069 students within 96 schools, we have 
measures of achievement and a verbal reasoning ability level from 1 to 3. In addition, 
the data contain the percent of students within each school who are eligible to 
participate in a free meal program. 


The input is: 
USE ILEA 


PLOT ACH*PFSM / GROUP-VRA OVERLAY COLOR-2,1,3, 
SMOOTH-LINEAR SHORT 


The output is: 


a 
x 
x 
x 
s 
4 
x 


ee 
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We begin by fitting a fixed effects only model that includes an intercept anda slope for 
each level of VRA. 


The input is: 


MIX 

USE ILEA 

RESET 

CATEGORY VRA / EFFECT 

MODEL ACH=INTERCEPT+PFSM+VRA+VRA*PFSM 
PLENGTH SHORT 

ESTIMATE 


The output is: 


Effects coding used for categorical variables in model 
Terms in the analysis and names of design matrix columns used for those terms: 


VRA 

FXD3 (1) FXD3 (2) 

VRA * PFSM 

FXD4 (1) FXD4 (2) 

Perform 10 EM iterations 
0 random terms 

6 fixed terms 


Numbers of Observations 


Level 2 observations : 2069 
Level 1 observations : 2069 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 
ACH 1.000 64.000 20.961 12.282 
INTERCEPT 1.000 1.000 1.000 0.000 
PFSM 10.760 70.320 31.826 11.636 
FXD3(1) -1.000 1.000 0.109 0.655 
FXD3 (2) -1.000 1.000 0.393 0.755 
FXD4 (1) -64.000 70.320 2.761 21.770 
FXD4 (2) -64.000 70.320 12.605 26.696 


Starting Values 


Covariates: 

25.141 70.153 12.810 72.219 -0.099 0.035 
Residual: 

106.861 


Final Results - MML Estimates 


EM Iterations 10 
Fisher Iterations 2 
Total Iterations 12 


Log Likelihood : -7765.476 
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Variable Estimate Standard Er: 
INTERCEPT Tere us : 
PFSM -0.153 0. 
FXD3 (1) 12.810 1. 
FXD3 (2) -2.219 0. 
FXD4 (1) -0.099 0. 
FXDA (2) 0.035 0. 
Residual Variance 
Estimate Standard Error Zz p-value 
106.551 3.313 32.164 0.000 
Correlation of the MML Estimates of the Fixed Terms 
i 1 2 3 4 5 6 
| INTERCEPT PFSM FXD3 (1) FXD3(2) PxD4 (1) FXD4 (2) 
PI T | IW cL EEUU d 
1  INTERCEPT | 1.000 
2 PFSM i -0.942 1.000 
3 FXD3 (1) H -0.006 -0.040 1.000 
4 FXD3 (2) H -0.480 0.461 -0.248 1.000 
5 FXDA (1) 1 -0.038 0.085 -0.943 0.257 1.000 
6 FXD4 (2) i 0.464 -0.498 0.267 -0.940 -0.308 1.000 


SYSTAT renames categorical variables according to whether they are fixed (FXD)or 
random (RND). To this name, an integer designating the position of the variable in the 
MODEL command (or the RANDOM command for random effects) is appended. Thus, 
all instances of FXD3 refer to VRA and instances of FXD4 refer to the VRA*PFSM 
interaction. 

SYSTAT recodes a categorical variable having k levels into k-1 dummy variables, 
using subscripts to denote the contrast represented by the variable. Verbal reasoning 
ability has three levels, requiring the generation of two dummy variables. Recall that 
in effect coding, the highest category serves as the reference category. As a result, 
FXD3(1) represents the contrast between the first and third levels of VRA. FXD3(2) 
represents the contrast between the second and third VRA levels. 


The effects of PFSM, VRA, and the interaction all appear significant. However, the 
slope for the second level of VRA does not appear to differ from the slope for the third 


level (p = 0.1995). 


Random Intercepts 


Instead of fitting a single line to each VRA level, we can account for the clustering of 


students within schools by including a random intercept. 
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The input is: 
MIX 
USE ILEA 
RESET 
CATEGORY VRA / EFFECT 
MODEL ACH=PFSM+VRA+VRA*PFSM 
RANDOM INTERCEPT 
IDENTIFIER SCHOOL 
SAVE RESIDUALS1 / RESID 
PLENGTH SHORT 
ESTIMATE 

The output is: 


Effects coding used for categorical variables in model 
Terms in the analysis and names of design matrix columns used for those terms: 


VRA 

FXD2 (1) FXD2 (2) 

VRA * PFSM 
FXD3 (1) FXD3 (2) 

Perform 10 EM iterations 
1 random terms 

5 fixed terms 


Numbers of Observations 


Level 2 observations : 96 
Level 1 observations : 2069 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 


FXD3 (2) -64.000 70.320 12.605 26.696 


Starting Values 


Mean: 
25.141 
Covariates: 
-0.153 12.810 -2.219 -0.099 0.035 
Variance Terms: 
21.372 
Residual: 
106.861 


Final Results - MML Estimates 


EM Iterations : 10 
Fisher Iterations : 4 
Total Iterations : 14 


Log Likelihood t 732,113 
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Variable Estimate Standard Error 


INTERCEPT 25.832 1.179 
PFSM -0.171 0.034 


FXD2 (1) 1,072 

FXD2 (2) 0.894 

FXD3 (1) 0.033 

FXD3 (2) 0.026 
Residual Variance 

Estimate Standard Error Z p-value 


98.136 3.122 31.433 0.000 


Random-Effect Variance & Covariance Term(s) 
Estimate 


i 1 
INTERCEPT 


1 INTERCEPT | 8.947 


1 INTERCEPT | 2.021 


p-value 


1 INTERCEPT | 0.000 


Note: p-values are 2-tailed except for those associated with variances, 


which are 1-tailed. 
Calculation of the Intracluster Correlation 


: 98.136 


Residual Variance AP 
81947 / (8.947 + 98.136) = 0.084 


Cluster Variance 
Intracluster Correlation 


Correlation of the MML Estimates of the Fixed Terms 


3 4 5 
FXD2(1)  FXD2(2) FXD3(1)  FXD3(2) 


H 
INTERCEPT 


H INTERCEPT 
2 PFSM 

1.000 
3 FXD2 (1) 

-0.255 1.000 
5 ENDS (1) -0.944 0.265 1.000 

-0. -0.318 1.000 

6 FXD3(2) 0.276 0.941 0 
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Correlation of the MML Estimates of Variance-Related Terms 


1 2 
VarCovl Residual 


i 1.000 
2 Residual | -0.075 1.000 


Empirical Bayes Estimates 


30 
03 
T 
028 
£ 1 
E 8 
E: 
10 041 p 
% 2 20 ado 
INTERCEPT 


The consequences of fitting the random intercept model can be displayed by plotting 
the predicted values. For plotting purposes, we need to recreate the original categorical 
variable, VRA, from the corresponding dummy variables. 


The input is: 
USE RESIDUALS1 
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LET VRA=1 

IF FXD2(1) <> 1 AND FXD2(2)=1 THEN LET VRA=2 

IF FXD2(1)=-1 AND FXD2(2)=-1 THEN LET VRA=3 

BEGIN 

PLOT PRED1*PFSM / OVERLAY GROUP=VRA, 
COLOR=2,1,3 YLAB='Predicted ACH' 

PLOT PREDO*PFSM / OVERLAY GROUP=VRA SIZE=0, 
SMOOTH=LINEAR SHORT COLOR=2,1, 3, 
LEGEND=NONE YLAB='' 

END 
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The output is: 


wN- < 


In the fixed effect model, every predicted value would lie on the lines in the plot. The 
random intercept allows each school its own regression line, and thus the predicted 
values vary by school. 


Predicted Values and Confidence Bands 


The previous plot showed the scatter of predicted values around the average regression 
line. The regression line for each school generated the predicted values. Can we see 
these lines? 

In theory, we could construct regression line for each school using the empirical 
Bayes estimates of the random effects. However, displaying 96 regression lines in a 
plot is probably a tad overwhelming. Instead, we will take advantage of the normality 
of the Bayes estimates to create confidence bands around the average regression line. 

The variance of the intercepts equals 8.9467, resulting in a standard deviation of 
2.9911. Normality implies that approximately 97% of the regression lines lie within 
two standard deviations of the average line. We can create these boundaries and 
display them in a plot with the predicted values. We use a multiplot to prevent 
cluttering a single plot with nine lines. 
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The input is: 


USE RESIDUALS1 

LET UPPER=PRED0+5 . 9822 

LET LOWER=PREDO-5.9822 

LET VRA=1 

IF FXD2(1) <> 1 AND FXD2(2)=1 THEN LET VRA=2 
IF FXD2(1)=-1 AND FXD2(2)=-1 THEN LET VRA=3 


BEGIN 

PLOT UPPER*PFSM / MULTIPLOT GROUP=VRA SMOOTH=LINEAR, 
SHORT YLAB='' COLOR=RED SIZE=0, 
FILL=1,0,0 LEGEND=NONE YMIN=0 YMAX=50, 
FTITLE=OFF 

PLOT LOWER*PFSM / MULTIPLOT GROUP=VRA SMOOTH=LINEAR, 
SHORT YLAB='' COLOR=RED SIZE=0, 
FILL=1,0,0 LEGEND=NONE YMIN=0 YMAX=50, 
FTITLE=OFF 

PLOT PREDO*PFSM / MULTIPLOT GROUP-VRA SMOOTH-LINEAR, 
SHORT YLAB-'' COLOR=BLUE SIZE=0, 
FILL=1,0,0 LEGEND=NONE YMIN=0 YMAX=50, 
FTITLE=OFF 

PLOT PRED1*PFSM / MULTIPLOT GROUP=VRA FILL=1,0,0, 
LEGEND=NONE YMIN=0 YMAX=50, 
YLAB='Predicted ACH' 


END 


The output is: 


HOV Peppe 


Sto e » € 99 9$ 
PFSM PFSM 


Example 3 A 
Longitudinal Data in Mixed Regression 


Riesby et al. (1977) studied the relati 


levels in plasma in 66 depressed patients classified as either endogenous or 


onship between desipramine and imipramine 
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nonendogenous. After receiving a placebo for one week, the researchers administered 
a dose of imipramine each day for four weeks, recording the imipramine and 
desipramine levels at the end of each week. At the beginning of the placebo week and 
at the end of each week (including the placebo week), patients received a score on the 
Hamilton depression rating scale. Did the depression score change over time 
differently for each group of patients (endogenous vs nonendogeneous)? 

A plot of the raw data often reveals general trends before fitting a model. 


The input is: 


MIX 

USE RIESBY 

BEGIN 

CATEGORY WEEK 

DENSITY HAMD * WEEK / BOX TICK=INDENT LOC=SIN, 0IN 

PLOT HAMD*WEEK / OVERLAY GROUP=ID LINE SIZE= 0, 
LEGEND-NONE TICK=INDENT LOC--1IN,OIN 

CATEGORY 

END 


The output is: 


o 1 


2 3 4 5 
WEEK 
The plot on the left, often referred to as a spaghetti plot, indicates a general decline in 


the Hamilton depression scores over time. In addition, the boxplots on the right 
demonstrate an increase in the variance of the depression scores over time. 
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Time as a Linear Effect 


One potential model for the Riesby data includes a different intercept for each patient, 
as well as a linear change in depression score over time. 


The input is: 


MIX 

USE RIESBY 

RESET 

MODEL HAMD 

IDENTIFIER ID 

RANDOM INTERCEPT WEEK 
SAVE RESIDUALS1 / RESID 
PLENGTH SHORT 

ESTIMATE 


The output is: 


Perform 10 EM iterations 
2 random terms 
0 fixed terms 


Numbers of Observations 


66 
375 


Level 2 observations 
Level 1 observations 


Descriptive Statistics for all variables 


INTERCEPT 
WEEK 


Starting Values 


Mean: 
23.603 -2.405 


Variance Terms: 
35.400 0.000 17.700 


Residual: 
35.400 


Final Results - MML Estimates 


EM Iterations d 10 
Fisher Iterations : 4 
Total Iterations + 14 
Log Likelihood E -1109.519 


WEEK 


Residual Variance 
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12.217 1.107 11.036 0.000 


Random-Effect Variance & Covariance Term(s) 


Estimate 


i 12.629 
2 WEEK H -1.421 2.079 


+ 
1 INTERCEPT | 3.467 


2 WEEK ' 1.026 0.504 
Z 
H 1 2 
! INTERCEPT WEEK 


1 INTERCEPT | 3.643 
2 WEEK H -1.385 4.124 
p-value 
i 1 2 
| INTERCEPT WEEK 
aaa ssn de 
1 INTERCEPT | 0.000 
2 WEEK i 0.166 0.000 


Note: p-values are 2-tailed except for those associated with variances, 
which are l-tailed. 


Random-Effect Covariances Expressed as Correlations 


2 
WEEK 
1 INTERCEPT | 
2 WEEK ; 1.000 
Correlation of the MML Estimates of the Fixed Terms 
2 
INTERCEPT WEEK 


1 INTERCEPT 
WEEK 
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Corr 


INTERCEPT 


WEEK 


Using the file of the residuals, we 


Mixed Regression 


elation of the MML Estimates of Variance-Related Terms 


H 2 3 4 
VarCovl  VarCov2 VarCov3 Residual 


VarCov2 -0.590 1.000 

VarCov3 0.220 -0.588 1.000 

Residual -0.180 0.169  -0.140 1.000 
Empirical Bayes Estimates 


can display the overall average effect, as well as the 


individual trends. 


The input is: 


USE RESIDUALS1 


BEGIN 
DOT HAMD PRED1 *WEEK / OVERLAY TI 


Trend', 


CK-INDENT TITLE- 'Average 


LEGEND-2.5IN, 3.21N 


LLABEL-'Observed', 'Predicted' T 
DRAW BOX / LOC-2.3IN, =.7IN WIDTH 1.3IN 


TICK-INDENT YLAB='' 
por ee ; P VERAY GROUP=ID SMOOTH=LINEAR SHORT, 


VE! 
scs ed j O GEND=NONE SIZE=0 TICK=INDENT, 
YLAB='Hamilton Depression Score', 
TITLE= ' Subject Trends' LOC=6IN, OIN 
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The output is: 


Average Trend Subject Trends 


8 8 5 


Hamilton Depression Score 
3 


a OE 
WEEK 


Overall, we see a general decline in scores. Individually, we find a few subjects who 
actually increased in depression score. 

The preferred method of establishing the significance of a random effect involves a 
log-likelihood comparison between two models: one in which the effect is random and 
another in which the effect is fixed. The log-likelihood for the latter model (not shown) 
equals -1142.5944. The two models differ in the inclusion of the variance for WEEK 
and the covariance between WEEK and the intercept. Thus, according to the log- 
likelihood test, the difference between the log-likelihoods times -2 follows a chi- 
square distribution with two degrees of freedom. This value equals 66.15 and is 
significant. WEEK should be treated as a random effect. 


Including Independent Variables 


To a model with a linear effect of time, we can add the effect of diagnosis to look for 
differences due to this factor. 


The input is: 


MIX 

USE RIESBY 

RESET 

MODEL HAMD = ENDOG WEEK*ENDOG 
IDENTIFIER ID 

RANDOM INTERCEPT WEEK 

SAVE RESIDUALS1 / RESID 
ESTIMATE 
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We include the interaction between ENDOG and WEEK to allow a separate trend for 
each type of diagnosis. 
The output is: 


Terms in the analysis and names of design matrix columns used for those terms 
WEEK * ENDOG 

FXD1 

Perform 10 EM iterations 

2 random terms 

2 fixed terms 


Numbers of Observations 


Level 2 observations = 66 
Level 1 observations : 375 


Descriptive statistics for all Variables 


HAMD 
INTERCEPT 
WEEK 
ENDOG 
FXD1 


starting Values 


Mean: 

22.518 -2,378 
Covariates: 

1.974 -0.045 
Variance Terms: 

34.721 0.000 17.361 
Residual: 


34.721 


Final Results - MML Estimates 


EM Iterations E 10 
Fisher Iterations : 4 
Total Iterations + 14 
Log Likelihood =? -1107.465 
Variable Estimate deine 


INTERCEPT 22.476 a 
WEEK -2.366 0.312 -7.587 0:009 
ENDOG 1.988 1.069 1-860 ages 
FXDl -0.027 0.419 -0.064 ; 


Residual Variance 


Estimate Standard Error 


11.037 0.000 


Random-Effect Variance & Covariance Term(s) 


Estimate 


1 
INTERCEPT WEEK 


i1 INTERCEPT | 11.641 
2 WEEK NEC cos 
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Standard Error 
i 1 2 
| INTERCEPT WEEK 
——— auro. tom sna e —— 
1 INTERCEPT | 3.296 
2 WEEK i 1.003 0.504 
z 
H 1 2 
| INTERCEPT WEEK 
a No 
1 INTERCEPT | 3.531 
2 WEEK i 71.397 4.123 
p-value 


H 1 2 
} INTERCEPT WEEK 
mp 
1 INTERCEPT | 0.000 
2 WEEK H 0.162 0.000 


Note: p-values are 2-tailed except for those associated with variances, 
which are 1-tailed. 


Random-Effect Covariances Expressed as Correlations 


H 1 2 
} INTERCEPT WEEK 


il 


1 INTERCEPT | 1.000 
WEEK 


2 t -0.285 1.000 
Correlation of the MML Estimates of the Fixed Terms 
f 1 2 3 4 
i INTERCEPT WEEK ENDOG — FXDl 
Pages LEE tiic a NM e Ur n 
1 1.000 
2 70.451 1.000 
3 70.743 0.335 1.000 
4 0.335  -0.743 -0.457 1.000 
Correlation of the MML Estimates of Variance-Related Terms 
H 2 
| VarCovi VarCov2  Vartovi  Residuas 
X pce ME ie ae ir m m 
1 1.000 
2 -0.601 1.000 
3 VarCov3 0.229 -0.598 1.000 
4 Residual | -0.189 0.173 -0.140 1.000 
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The parameter estimates suggest that ENDOG may have an effect (p =.06), but the two 
groups do not differ in their rate of change in depression (p =.95). We use scatterplots 
to display the results of this model. 


The input is: 


USE RESIDUALS1 


BEGIN 
PLOT PRED1*WEEK / OVERLAY GROUP=ENDOG, 
TICK= 


INDENT, 
YLAB-'Hamilton Depression Score', 
LTITLE-'' LLABEL='Non-Endog.', *Endog', 
YMIN-0 YMAX-40 
OVERLAY GROUP-ENDOG SMOOTH=LINEAR SHORT, 
SIZE-0 TICK-INDENT LEGEND-NONE, 
YLAB-'' YMIN-0 YMAX-40 


PLOT PREDO*WEEK / 


END 
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The output is: 


5 
td 


E- 


Hamilton Depression Score 


e 


The two lines are essentially parallel. 


Time as a Quadratic Effect 


In this example, we fit a quadratic effect of time to the Riesby data, ignoring the 
diagnosis of the patients. 


The input is: 


MIX 

USE RIESBY 

RESET 

MODEL HAMD 

IDENTIFIER ID 

RANDOM INTERCEPT WEEK WEEK*WEEK 
SAVE RESIDUALS1 / RESID 
ESTIMATE 
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Terms in the analysis and names of design matrix columns used for those terms: 


WEEK * WEEK 

RND2 

Perform 10 EM iterations 
3 random terms 

0 fixed terms 


Numbers of Observations 


Level 2 observations : 66 
Level 1 observations : 375 


Descriptive Statistics for all Variables 


Standard 


Variable Minimum Mean Deviation 


HAMD 0.000 17.637 7.190 
INTERCEPT 1.000 1.000 0.000 
WEEK 0.000 2.480 1.683 
RND2 0.000 8.976 8.734 


Starting Values 


Mean: 

23.759 -2.636 0.046 
Variance Terms: 

35.482 0.000 17.741 0.000 0.000 17.741 
Residual: 

35.482 
Final Results - MML Estimates 
EM Iterations 1 10 
Fisher Iterations : 5 
Total Iterations : 15 
Log Likelihood : -1103.824 
Variable Estimate Standard Error z 
INTERCEPT 23.760 0.552 43.039 0.000 
WEEK -2.633 0.479 -5.496 0.000 
RND2 0.051 0.088 0.583 0.560 


Residual Variance 


Estimate Standard Error 


Random-Effect Variance & Covariance Term(s) 


Estimate 


1 INTERCEPT | 10.440 
2 WEEK H -0.915 6.638 
3  RND2 -0.112  -0.936 0.194 


Standard Error 
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INTERCEPT WEEK RND2 
1 INTERCEPT 3.579 
2 WEEK 2.418 2.746 
3  RND2 0.421 0.484 0.094 
z 
1 INTERCEPT 2.917 
2 WEEK i -0.379 2.418 
3  RND2 i -0.266 -1.933 2.063 
p-value 
1 2 3 
INTERCEPT WEEK RND2 


1 
H 
i 

ee ee foamed AS 
1 
i 
D 


1 INTERCEPT 0.002 
2 WEEK 0.705 0.008 
3  RND2 0.790 0.053 0.020 


Note: p-values are 2-tailed except for those associated with variances, 
which are l-tailed. 


Random-Effect Covariances Expressed as Correlations 


1 1 2 3 
| INTERCEPT WEEK RND2 
e A is 
1 INTERCEPT | 1.000 
2 WEEK i -0.110 1.000 
3  RND2 ! -0.079 -0.826 1.000 


Correlation of the MML Estimates of the Fixed Terms 


i 1 2 3 
| INTERCEPT WEEK RND2 


10 
0.299 -0.902 1.000 
Correlation of the MML Estimates of Variance-Related Terms 


1 2 3 4 5 6 
VarCovl VarCov2 VarCov3  VarCov4 VarCov5  VarCov6 


i 
* 
1  VarCovi | 1.000 
2  VarCov2 | -0.603 1.000 
3  varCov3 | 0.255 -0.612 1.000 
4  VarCov4 | 0.428 -0.909 0.579 1.000 
5 VarCovS ; -0.201 0.518 -0.952 -0.544 1.000 
6 VarCov6 | 0.158 -0.401 0.833 0.445 -0.953 1.000 
7 Residual | -0.263 0.280 -0.305 -0.244 0.320 70.333 


Correlation of the MML Estimates of Variance-Related Terms (contd...) 


H 7 
Residual 


VarCovl 
VarCov2 
VarCov3 
VarCov4 
VarCov5 
VarCov6 
Residual ; 1.000 


Youswnes 
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Empirical Bayes Estimates 


Although dividing the parameter estimate for the quadratic effect by its standard error 
suggests that a quadratic effect is not needed, the log-likelihood ratio test suggests 
otherwise (-2*ALL = 11.4 on 4 degrees of freedom). The quadratic effects are plotted. 


The input is: 


USE RESIDUALS1 

PLOT PRED1*WEEK / OVERLAY GROUP=ID SMOOTH=SPLINE SHORT, 
LEGEND=NONE SIZE=0 TICK=INDENT, 
YLAB='Hamilton Depression Score' 


The output is: 
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Autocorrelated Errors 


To illustrate the inclusion of autocorrelated errors in longitudinal data, we add a first- 
order autoregressive structure to the model containing a linear effect of time and an 
effect of diagnosis. 


The input is: 


MIX 

USE RIESBY 

RESET 

MODEL HAMD - ENDOG WEEK*ENDOG 
IDENTIFIER ID 

RANDOM INTERCEPT WEEK 

AUTO WEEK 

SAVE RESIDUALS1 / RESID 
ESTIMATE / NEM=0 


The output is: 


Terms in the analysis and names of design matrix columns used for those terms: 
WEEK * ENDOG 

FXD1 

Perform 0 EM iterations 

2 random terms 

2 fixed terms 

Autocorrelated Error Structure: AR(1) 


Numbers of Observations 
Level 2 observations : 66 
Level 1 observations : 375 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 
HAMD 0.000 39.000 17.637 7.190 
INTERCEPT 1.000 1.000 1.000 0.000 
WEEK B . 2.480 1.683 
ENDOG 0.000 1.000 0.547 0.498 
FXD1 0.000 5.000 1.352 1.746 


Starting Values 


Mean: 

22.518 -2.378 
Covariates: 

1.974 -0.045 


Variance Terms: 
34.721 0.000 17.361 

Residual: f 
34.721 

Auto Terms: l 
0.200 | 


Final Results - MML Estimates 
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EM Iterations 0 
Fisher Iterations 14 
Total Iterations 14 
Log Likelihood -1103.419 
Variable Estimate standard Error z p-value 
INTERCEPT 22,462 0.787 28.545 0.000 
WEEK -2.328 0.303 -7.688 0.000 
ENDOG 1.870 1.060 1.764 0.078 
FXD1 -0.016 0.408 -0.040 0.968 


Residual Variance 


Estimate Standard Error Z p-value 


Autocorrelation Term(s) 


0.371 0.122 3,042 0.002 


Random-Effect Variance & Covariance Term(s) 


Estimate 


1 2 

INTERCEPT WEEK 
INTERCEPT | 3.901 

2 WEEK i 0.340 


Standard Error 


2 WEEK 


1 INTERCEPT 0.735 
2 WEEK i 0.269 2.199 
p-value 
1 2 
INTERCEPT WEEK 
1 INTERCEPT 0.231 


0.788 0.014 
iled except for those associ 


2 WEEK 
Note: p-values are 2-ta ated with variances, 
which are l-tailed. 


Random-Effect Covariances Expressed as Correlations 


t E 
| INTERCEPT 
1.000 
0.152 1.000 


1 INTERCEPT 
2 WEEK 


Correlation of the MML Estimates of the Fixed Terms 
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i 1 

! INTERCEPT 
-—-—-------- + 
INTERCEPT | 
WEEK H 
ENDOG t 
FXD1 D 


2 3 4 
WEEK ENDOG FXD1 
1.000 
0.332 1.000 
-0.742 -0.454 1.000 


Correlation of the MML Estimates of Variance-Related Terms 


INTERCEPT 


The log-likelihood test: 
-2*(-1107.465+1103.419) = 8.1 


WEEK 


i 1 2 3 4 5 
| VarCovl VarCov2 VarCov3 Residual AutoCorl 

—ÁÁ—À 2 qui A MO SERIO. a Suec saevit 

VarCovl | 1.000 

VarCov2 | -0.788 1.000 

VarCov3 | 0.564 70.741 1.000 

Residual | -0.700 0.600 -0.530 1.000 

AutoCorl | -0.764 0.610 -0.539 0.685 1.000 

Empirical Bayes Estimates 
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has one degree of freedom due to the inclusion of the autoregressive parameter. This 
value is significant, suggesting the need for the autocorrelation structure. This structure 
has the following form: 


1.00 

0.37 1.00 

0.14 0.37 1.00 

0.05 0.14 0.37 1.00 


0.02 0.05 0.14 0.37 1.00 
0.01 0.02 0.05 0.14 0.37 1.00 
0.002 0.01 0.02 0.05 0.14 0.37 1.00 


Notice that although the inclusion of this matrix does not affect the fixed parameter 
estimates to any significant degree, the variances of the random parameters do change. 
The largest change occurs in the variance of the intercept, which drops from 11.6412 


to 3.9009. 


Example 4 
Multivariate Layout for Longitudinal Data 


In this example, we analyze data having multivariate layout from a study by Morrison 
and Zeppa (1963). In this study, mongrel dogs were divided dogs into four groups of 
four. The groups received different drug treatments. The dependent variable, blood 

histamine in mg/mL, was then measured at four times after administration of the drug. 
The data are incomplete, since one of the dogs is missing the last measurement. We use 


a repeated-measures scatterplot to display. 


The input is: 
USE HISTAMINE 
PLOT HISTAMINE1. -HISTAMINE4 / OVERLAY REPEAT, 
GROUP=DOG, 
LINE LEGEND=NONE 
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The output is: 


The variance in the histamine levels varies over time. In an effort to stabilize the 
variance, we apply a log-transformation. 


USE HISTAMINE 

let LNHIST1=LOG(HISTAMINE1) 

let LNHIST2=LOG (HISTAMINE2) 

let LNHIST3-LOG (HISTAMINE3) 

let LNHIST4=LOG (HISTAMINE4) 

PLOT LNHIST1..LNHIST4 / OVERLAY REPEAT GROUP-DOG, 
LINE LEGEND-NONE 
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LNHIST! LNHIST2 LNHIST3 LNHISTA 
Trial 


The logged histamine levels now exhibit a similar spread of values at each 
measurement occasion. Subsequent analyses will use the logged values. 


Random Intercept with Fixed Categorical Effects 


To study the effects of the four drugs over time, we include the drug, a measure of time, 
and their interaction as fixed effects in a mixed regression model, To account for the 
dependencies due to taking repeated measurements on each dog, we include the dog as 
a random effect. 

The dependent variable, histamine level, does not appear as a variable in the data file, 
HISTAMINE. Instead, the data file uses a multivariate layout, recording the histamine 
level across four variables representing time. To rearrange this data into a hierarchical 
structure, we use CONVERT. 
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The input is:. 


USE HISTAMINE 

LET LNHIST1=LOG (HISTAMINE1) 

LET LNHIST2=LOG (HISTAMINE2) 

LET LNHIST3=LOG (HISTAMINE3) 

LET LNHIST4=LOG (HISTAMINE4) 

MIX 

RESET 

CONVERT HIST=LNHIST1. . LNHIST4 

CAT DRUG TRIAL / EFFECT 

MODEL HIST = DRUG + TRIAL + DRUG*TRIAL 
IDENTIFIER DOG 

RANDOM INTERCEPT 

SAVE RESIDUALS1 / RESID 

PLENGTH SHORT 

ESTIMATE / REPAR=OFF NREC=1 NEM=0 


Notice that the variable TRIAL, the time variable, does not appear in the original data 
file. CONVERT creates this variable. 


The output is: 


Effects coding used for categorical variables in model 

Terms in the analysis and names of design matrix columns used for those terms: 
DRUG 

FXD1 (1) FXD1 (2) FXD1 (3) 

TRIAL 

FXD2 (1) FXD2 (2) FXD2 (3) 

DRUG * TRIAL 

FXD3 (1) FXD3 (2) FXD3 (3) FXD3 (4) FXD3 (5) FXD3 (6) FXD3 (7) FXD3 (8) FXD3 (9) 
Perform 0 EM iterations 

1 random terms 

15 fixed terms 


Numbers of Observations 


Level 2 observations : 16 
Level 1 observations : 63 


Descriptive Statistics for all Variables 


Standard 
Variable Minimum Maximum Mean Deviation 
HIST -3.912 1.141 =1.977 1.172 
INTERCEPT 1.000 1.000 1.000 0.000 
FXD1 (1) -1.000 1.000 0.000 0.718 
FXD1(2) -1.000 1.000 -0.016 0.707 
FXD1 (3) 71.000 1.000 0.000 0.718 
FXD2 (1) 71.000 1.000 0.016 0.707 
FXD2 (2) -1.000 1.000 0.016 0.707 
FXD2 (3) -1.000 1.000 0.016 0.707 
FXD3(1) -1.000 1.000 0.000 0.508 
FXD3(2) -1.000 1.000 0.000 0.508 
FXD3(3) 71.000 1.000 0.000 0.508 
FXD3 (4) 71.000 1.000 0.016 0.492 
FXD3(5) 71.000 1.000 0.016 0.492 
FXD3(6) -1.000 1.000 0.016 0.492 
FXD3 (7) -1.000 1.000 0.000 0.508 
FXD3 (8) -1.000 1.000 0.000 0.508 
FXD3(9) 71.000 1.000 0.000 0.508 


11-477 


Mixed Regression 


starting Values 


Mean: 
-1.984 

Covariates: 
-0.109 -0.487 1.091 -0.728 0.474 0.207 
-0.069 0.458 -0.113 0.680 -0.486 -0.189 
-1.399 0.551 0.512 

Variance Terms: 
0.115 

Residual: 
0.577 


Total Number of Level-2 Units = 16 
Data for Level-2 Unit 1 which has 4 observations nested within 


Dependent Variab le Vector 


i 1 
1 | -3.219 
2 | -1.609 
3 | -2.303 
4 | -2.526 


Random-effect Design Matrix 


D 1 


1 
2 
3 
4 


m 
o 
o 
o 


0.000 0.000 1.000 0.000 

0.000 0.000 0.000 1.000 0.000 0.000 1.000 0.000 
0.000 0.000 0.000 0.000 1.000 0.000 0.000 1.000 
0.000 0.000  -1.000 -1.000 -1.000  -1.000 -1.000 -1.000 


Final Results - MML Estimates 


EM Iterations E 0 
Fisher Iterations : 5 
Total Iterations : 5 
Log Likelihood : -23.789 
Variable Estima Standard Error 
INTERCEPT 0.155 
FXD1 (1) 0.269 
FXD1 (2) 0.269 
FXD1(3) 0.269 
FXD2 (1) 0.050 
FXD2 (2) 0.050 
FXD2 (3) 0.050 
FXD3 (1) 0.08 


FXD3 (2) 
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FXD3 (3) 70.107 0.087 -1.237 0.216 
FXD3 (4) 0.664 0.088 7.569 0.000 
FXD3 (5) -0.502 0.088 75.731 0.000 
FXD3 (6) 70.206 0.088 72.349 0.019 
FXD3(7) 71.393 0.087 -16.084 0.000 
FXD3 (8) 0.556 0.087 6.422 0.000 
FXD3(9) 0.518 0.087 5.980 0.000 


Residual Variance 
Estimate Standard Error z 
0.053 0.011 4.848 


Random-Effect Variance & Covariance Term(s) 
Estimate 


1 INTERCEPT | 0.003 


Note: p-values are 2-tailed except for those associated with variances, 
which are l-tailed. 


Calculation of the Intracluster Correlation 


Residual Variance 
Cluster Variance 


Correlation of the MML Estimates of the Fixed Terms 


; 1 2 3 4 5 6 
| INTERCEPT FXD1 (1) FXD1 (2) FXD1 (3) FXD2 (1) FXD2 (2) 
1 INTERCEPT ; 1.000 
2  FXD1(1) 1 -0.001 
Li FXD1 (2) i 0.002 
4 FXD1(3) ! -0.001 
5  FXD2(1) i 70.003 
6  FXD2(2) 1 -0.003 1.000 
7  FXD2(3) H -0.003 -0.321 
8  FXD3(1) i 0.002 70.005 
9  FXD3(2) i 0.002 -0.005 
10  FXD3(3) 1 0.002 70.005 
11  FXD3(4) i -0.005 0.016 
12  FXD3(5) i -0.005 0.016 
13  FXD3(6) D 70.005 0.016 0.016 
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Mixed Regression 
14  FXD3() | 0.002  -0.001 0.003 -0.001 -0.005 -0.005 
15 FXD3(8) | 0.002 -0.001 0.003 -0.001 -0.005 -0.005 
16  FXD3(9) | 0.002 -0.001 0.003 . -0,001  -0.005 -0.005 


Correlation of the MML Estimates of the Fixed Terms (contd...) 


12 


8 5 10 1i 
FXD3(1)  FXD3(2)  FXD3(3)  FXD3(4)  FXD3(5) 


1.000 

-0.329 1.000 

-0.329 -0.329 1.000 

-0.337 0.100 0.100 1.000 

. 0.100 70.337 0.100 -0.298 1.000 

13  FXD3(6) i 0,016 0.100 0.100 -0.337 -0.298 -0.298 
14  FXD3(7) | -0.005 -0.329 0.114 0.114 -0.337 0.100 
15  FXD3(8) i -0.005 0.114 -0.329 0.114 0.100 -0.337 
16  FXD3(9) 1 -0.005 0.114 0.114 -0.329 0.100 0.100 


Correlation of the MML Estimates of the Fixed Terms (contd...) 


i 13 14 15 16 
| FXD3(6)  FXD3(7)  FXD3(8) FXD3(9) 
1 INTERCEPT 
2  FXDl(1) t 
3 FXD1 (2) i 
4  FXDl(3) i 
5  FXD2(1) i 
6 | FXD2(2) i 
7 . FXD2(3) i 
8  FXD3(1) H 
9  FXD3(2) i 
10  FXD3(3) i 
11. FXD3(4) i 
12  FXD3(5) 1 
13  FXD3(6) t 
14  FXD3(7) H 
15 FXD3(8) t 
16  FXD3(9) i 


Correlation of the MML Estimates of Variance-Related Terms 


1.000 
0.100 1.000 
0.100 -0.329 1.000 
-0.337 -0.329 -0.329 1.000 


1  VarCovl 
2 Residual | 
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8 0.5 


3 2 
INTERCEPT 


DRUG is the first fixed effect variable included in the model so the coded variable for 
DRUG uses a root name of FXD/. The variable has four levels so three effect-coded 
variables are needed. Furthermore, TR/AL has four levels so three effect-coded 
variables are needed. The final fixed effect, the interaction, involves the crossing of 
two variables, each having four levels, so (4-1)*(4-1) effect-coded variables are 
needed. 

The output includes a listing of the converted data for the first level 2 unit. The 
dependent variable vector displays the log transformed histamine levels, The random- 
effect design matrix shows that the dog in question is the first. The covariate matrix 
corresponds to the fixed effects in the model as follows: 


m The first three columns represent DRUG. The first dog received drug 1, so the first 
column equals | and the next two equal 0. 


W The next three columns represent 7R/AL, comparing each measurement occasion 
to the last. 


m The final nine columns represent the interaction and result from crossing the first 
three columns with the second three. 


Looking at the parameter estimates for the fixed effects, we find that although some 
contrasts are not significant, each factor does exhibit a significant effect. This suggests 
that the histamine levels varied over time differently for each drug. The predicted 
values may shed some light on how these factors interact, 


11-481 


Mixed Regression 


Predicted Values 


The RESIDUALS] file contains the predicted values as well as the residuals. To 
illustrate the effects in the model, we create a multiplot of the predicted values with the 
original data. 


The input is: 


USE RESIDUALS1 / NONAMES 

IF FXD2(1)=1 THEN LET TRIAL=1 

IF FXD2(2)=1 THEN LET TRIAL=2 

IF FXD2(3)=1 THEN LET TRIAL=3 

IF FXD2(3)=-1 THEN LET TRIAL=4 

IF FXD1(1)=1 THEN LET DRUG=1 

IF FXD1(2)=1 THEN LET DRUG=2 

IF FXD1(3)=1 THEN LET DRUG=3 

IF FXD1(3)=-1 THEN LET DRUG=4 

BEGIN 

LINE PRED1*TRIAL / MULTIPLOT GROUP-DOG COL=4, 
YMIN=-5 YMAX-2 YLAB='' GROUPTITLE=OF 

PLOT HIST*TRIAL / MULTIPLOT GROUP=DOG COL=4, 
YMIN=-5 YMAX=2 YLAB='Ln (Histamine) ' 


END 
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The output is: 


[el 


m 
pl. xu 
RUBER 
HE | 


The multiplot clearly illustrates the interaction between DRUG and TRIAL. Those dogs 
receiving drugs 2 and 4, the second and fourth rows of the multiplot, show very little 
change in histamine level over time. The remainder of the dogs exhibited a sharp 
increase in histamine level, followed by a general decrease. 
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Residuals 


To examine the adequacy of the model, we focus on the residuals. We can assess their 
normality using a probability plot. 


The input is: 


USE RESIDUALS1 / NONAMES 

IF FXD2(1)=1 THEN LET TRIAL=1 

IF FXD2(2)=1 THEN LET TRIAL=2 

IF FXD2(3)=1 THEN LET TRIAL=3 

IF FXD2(3)=-1 THEN LET TRIAL=4 

IF FXD1(1)=1 THEN LET DRUG=1 

IF FXD1(2)=1 THEN LET DRUG=2 

IF FXD1(3)=1 THEN LET DRUG=3 

IF FXD1(3)=-1 THEN LET DRUG=4 

PPLOT RES1 / NORMAL SMOOTH=MIDRANGE 


The output is: 


N 


Normal( 0.0, 1.0) Quantile 
o 


È 


o 05 05 1.0 


00 
RES1 


h the residuals lie indicates that normality of the residuals 


The straight line along whic o 1 
other model assumptions by plotting the 


appears to be satisfied. We can examine 
residuals against the predicted values. 
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The input is: 
BEGIN 
PLOT RESI*PRED1 / OVERLAY GROUP-DRUG LOC--1IN,OIN, 
LEGEND-4.2IN,.1IN 
PLOT RES1*PRED1 / OVERLAY GROUP-TRIAL LOC-SIN,OIN, 
LEGEND=4 .2IN, .1IN 
END 
The output is: 
PL a JON 10 Ti» PPR 
ast o 4 05 P 4 
- oa s o As 
a o 
i oof prd 7 Wa ^ dh 
DRUG 4 TRIAL 
" joa ]o1 
ost > 05 H 
^ 3 A S 
a4 Ad 
EE ee O | 410! " ———Ü | pad. 
5 4 3 2 A 0 1 5 4 3 2 A 0 1 
PRED1 PRED1 


The residuals for the random intercept model are scattered randomly about zero. There 
appears to be no relation between the residuals and the predicted values. 

If we focus on the fixed effects corresponding to the plot symbols, we see a very 
small range over which the predicted values vary for both drugs 2 and 4. These drugs 
both resulted in relatively constant histamine levels over time. Furthermore, it appears 
that the variance of the residuals may be decreasing over trials. You may want to 
examine the effects of including an autocorrelation structure in the model. 


Computation 


Algorithms 


Mixed regression uses marginal maximum likelihood to estimate the parameters of the 
model. The procedure involves a combination of the EM algorithm and Fisher scoring. 
For details, see Hedeker and Gibbons (1996). 
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Acronym & Abbreviation 


Expansions 


A 

ABS - absolute value 

ACF - autocorrelation function 

ACOLOR - color axes 

ACS - arccosine 

ACT - actuarial life table 

AD test - Anderson Darling test 

ADDTREE - additive trees 

ADFG - asymptotically distribution free estimate 
biased, Gramian 

ADFU - asymptotically distribution free estimate 
unbiased 

ADJSEASON - seasonal adjustment 

AHMAX - maximum extent 

AHMIN - minimum extent 

AIC - Akaike information criterion 

AID - automatic interaction detection 

ALT - alternative 

ANCOVA - analysis of covariance 

ANGI - deviation of angles from north in a 
clockwise direction 

ANG2 - deviation of angles from horizontal (for 
3D models) 

ANG3 - tilt angle 

ANOVA - analysis of variance 
ANOVAHYPO - hypothesis tests in analysis of 
variance 

AR - autoregressive 

ARIMA - autoregressive integrated moving 
average 

ARL - average run length 
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ARMA - autoregressive moving average 
ARS - adaptive rejection sampling 
ASCII - American Standard Code for 
Information Interchange 

ASE - asymptotic standard error 

ASN - arcsine 

ATH - arc hyperbolic tangent 

ATN - arctangent 

AVERT - vertical extent 

AVG - average 


B 

BC - Bray-Curtis similarity measure 
BCa - Bias Corrected and accelerated 
BCF - Beta cumulative function 
BDF - Beta density function 
BETACORR - beta correction 

BIC - Bayesian information criterion 
BIF - Beta inverse function 

BMP - Windows bitmap 

BOF - beginning-of-file 

BOG - beginning-of-BY group 
BONE - Bonferroni 

BOOT - bootstrap 

BRN - Beta random number 


€ 

CART - classification and regression trees 
CBSTAT - column basic statistics 

CCF - Cauchy cumulative function 

CCF - cross-correlation function 

CDF - Cauchy density function 

cdf/CF - cumulative distribution function 
CDFUNC - coefficients for canonical variables 
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Acronyms 


CFUNC - coefficients for the classification 
functions 
CGM - Computer graphics metafile: binary or 
clear text 
CHAZ - cumulative hazard 
CHISQ - Chi-square distribution 
CHOL - Cholesky decomposition 
CI - confidence interval 
CIF - Cauchy inverse function 
CIM - confidence interval of mean 
CLASS - classification 
CLSTEM - stem and leaf plot for column 
CMeans - canonical scores of group means 
CMULTIVAR - multiple string variables 
COEF - coefficients 
COL/col - column 
COLPCT - Column percentages 
CONFIG - configuration 
CONT - Contingency coefficient 
CONV - convergence 
CORAN - correspondence analysis 
CORR - correlations 
CORRI - single correlation coefficient 
CORR2 - equality of two correlations 
COV - covariance 
Cp - process capability index 
CPL - process capability based on lower 
specification limit 
CPU - process capability based on upper 
specification limit 
Cpk-Process capability index for off-centered 
process 
CR - confidence region 
CRA - cost of response above UTL 
CRB - cost of response below LTL 
CRN - Cauchy random number 
CSCORE - canonical scores 
CSIZE - size of characters 
CSQ - Chi-square 
CSTATISTICS - column statistics 
CSV - comma separated values 


CUSUM - cumulative sum 

CUSUM HI - Upper cumulative sum 
CUSUM LO - Lower cumulative sum 
CV - coefficient of variation 

CVI - cross validation index 


D 

DBF - Dbase files 

DC - deciles of risk 

DECF - Double exponential cumulative function 
DEDF - Double exponential density function 
DEIF - Double exponential inverse function 
DENFUN - density function 

dep. - dependent 

DERN - Double exponential random number 
DET - determinant 

DEVI - deviates (observed values - expected 
values) 

DEXP - Double exponential distribution 

df - degrees of freedom 

DF - distribution function 

DHAT - estimated distance 

DIF - data interchange format 

DIM - dimension 

DISCRIM - discriminant analysis 

DIST - distance 

DIT - dot histogram 

DOE - design of experiments 

DOS - disc operating system 

DPMO - defects per million opportunities 
DPU - defects per unit 

DTA - Stata files 

DUCF - Discrete uniform cumulative function 
DUDF - Discrete uniform density function 
DUIF - Discrete uniform inverse function 
DUNIFORM - Discrete uniform 

DURN - Discrete uniform random number 
DWLS - distance weighted least-squares 


E 
ECF - Exponential cumulative function 
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EDF - Exponential density function 
EEXP - extreme value exponential 
EIF - Exponential inverse function 
EIGEN - eigenvalues 

ELAMBDA - exp(lambda) 

EM - expectation-maximization 

EMF - Windows enhanced metafile 
ENCE - Logit normal cumulative function 
ENDF - Logit normal density function 
ENIF - Logit normal inverse function 
ENORMAL - Logit normal 

ENRN - Logit normal random number 
EOF - end-of-file 

EOG - end-of-BY group 

EPS - Encapsulated postscript 

ERN - Exponential random number 
ES - exhaustive search 

ESS - error sum of squares 

EW - extreme value Weibull 

EWMA - exponentially weighted moving average 
EXP/exp - exponential/ expected 


F 

FAR - false-alarm rates 

FCF - F cumulative function 
FCOLOR - color foreground 

FDF - F density function 

FIF - F inverse function 

FINV - inverse of the F cumulative 
FITC - fitting distribution: continuous 
FITD - fitting distribution: discrete 
FITDIST - fitting distributions 
Flexibeta - flexible beta 

FPLOT - function plots 

FRN - F random number 

FTD - folded trellis detector 
FTDEV - Freeman-Tukey deviate 
FULLCOND - full conditional 
FUN - function 


G 


Acronyms 


GCF - Gamma cumulative function 
GCOR - groupwise correlation matrix 
GCOV - groupwise covariance matrix 
GCV - generalized cross validation 
GDF - Gamma density function 

GECF - Geometric cumulative function 
GEDF - Geometric density function 
GEIF - Geometric inverse function 
GEN - general Toeplitz structure 
GERN - Geometric random number 
GG - Greenhouse Geisser 

GIF - Gamma inverse function 

GIF - Graphics Interchange Format 
GLM - generalized linear models 
GLMHYPO - hypothesis tests in general linear 
model 

GLMPOST - post hoc estimate for repeated 
measures in general linear model 

GLS - generalized least-squares 

GMA - geometric moving average 

GN - Gauss-Newton method 

GOCE - Gompertz cumulative function 
GODF - Gompertz density function 
GOIF - Gompertz inverse function 
GORN - Gompertz random number 
GRN - Gamma random number 

GUCF - Gumbell cumulative function 
GUDF - Gumbell density function 
GUIF - Gumbell inverse function 
GURN - Gumbell random number 


H 

H & L - Hosmer and Lemeshow 

HC - heteroscedasticity-consistent 

HCF - Hypergeometric cumulative function 
HDF - Hypergeometric density function 
HF- Huynh-Feldt 

HGEOMETRIC - hypergeometric 

HIF - Hypergeometric inverse function 
HIST - histogram 

HKB - Hoerl, Kennard, and Baldwin 
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Acronyms 


H-L trace - Holding-Lawley trace 

HR - hit-rates 

HRN - Hypergeometric random number 
HSD - honestly significant differences 
HTERM - terms tested hierarchically 
HTML - hyper text markup language 
HYMH - hybrid Metropolis-Hastings 


I 

IF - Inverse cumulative distribution function 
IGAUSSIAN - inverse Gaussian 

IGCF - Inverse Gaussian cumulative function 
IGDF - Inverse Gaussian density function 
IGIF - Inverse Gaussian inverse function 
IGRN - Inverse Gaussian random number 
IIDMC - independently and identically 
distributed Monte Carlo 

IMPSAMPI - importance sampling integration 
IMPSAMPR - importance sampling ratio 
I-MR - individual and moving range 
Ind/indep - independent 

IndMH - Independent Metropolis-Hastings 
INDSCAL - individual differences scaling 
INITSAMP - initial sample 

INTEG FUN - integrated function 

IPA - iterated principal axis 

ITER - iterations 


J 

JACK - jackknife 

JCLASS - jackknifed classification 

JMP - JMP v3.2 data files 

JPEG/JPG - joint photographic experts group 


K 

K-M - Kaplan-Meier 

KNBD - kth nearest neighborhood 

KRON - Kronecker product 

K-S test - Kolmogorov-Smirnov test 

KSI - one sample Kolmogorov-Smirnov tests 
KS2 - two sample Kolmogorov-Smirnov tests 


L 

LAD - least absolute deviations 

LB - larger the better 

LCF - Logistic cumulative function 
LCHAZ - log cumulative hazard 

LCL - lower control limit 

LCONV - log-likelihood convergence criteria 
LDF - Logistic density function 

LGM - log gamma 

LGST - logistic 

LIF - Logistic inverse function 

L-L/LL - log likelihood 

LMS- least median of squares 
LMSREG - least median of squares regression 
LNCF - Lognormal cumulative function 
LNDF - Lognormal density function 
LNIF - Lognormal inverse function 
LNOR/LNORMAL - lognormal 

LNRN - Lognormal random number 

loc - location 

LOGI - one-parameter logistic (Rasch) 
LOG2 - two-parameter logistic 

LOGIT - logistic regression 
LOGITHYPO - hypothesis tests in logistic 
regression 

LOGLIN - loglinear modeling 

LR - likelihood ratio 

LRCHI - likelihood ratio chi-square 
LRDEV - likelihood ratio of deviate 
LRN - Logistic random number 

LS - least-squares 

LSD - least significant difference 

LSL - lower specification limit 

LSQ - least-squares 

LTAB - life tables 

LTL - lower tolerance limit 

LW - Lawless and Wang 


M 
MA - moving average 
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MAD - mean absolute deviation 

MAHAL - Mahalanobis distances 

MANCOVA - multivariate analysis of covariance 
MANOVA - multivariate analysis of variance 
MANOVAHYPO - hypothesis tests in 
MANOVA 

MANOVAPOST - post hoc estimate for repeated 
measures in MANOVA 

MAR - missing at random 

MAX - maximum 

MAXSTEP - maximum number of steps 

MCAR - missing completely at random 

MCMC - Markov Chain Monte Carlo 

MDPREF - multidimensional preference 

MDS - multidimensional scaling 

MIN - minimum 

M-H- Metropolis-Hastings 

MIS - number of missing values 

MIX - mixed regression 

MIXHIER - mixed regression for data having a 
hierarchical structure 

MIXMULTY - mixed regression for data having 
a multivariate structure 

ML - Maximum Likelihood 

MLA - maximum likelihood analysis 

MLE - maximum likelihood estimate 

MML - maximum marginal likelihood 

MRC - Multiple Regression and Correlation 

MS - mean squares 

MSE - mean square error 

MSIGMA - sigma measurement 

MT - Mersenne-Twister 

MTW - MINITAB v11 data files 

MU2 - Guttman's mu2 monotonicity coefficients 
MULTIVAR - multiple variables 

MW - minimum within sum of squares deviations 
MWL - maximum Wishart likelihood 


N 
NAR - non-stationary first-order autoregressive 
NB - nominal the best 


Acronyms 


NBB - nominal-the-best: bilateral tolerance 
NBCF - Negative binomial cumulative function 
NBD - number of active bounds on parameter 
values 

NBDF - Negative binomial density function 
NBIF - Negative binomial inverse function 
NBINOMIAL - Negative binomial 

NBRN - Negative binomial random number 
NBU - nominal-the-best: unilateral tolerance 
NCAT - number of categories 

NCF - Binomial cumulative function 

NCOL - number of columns 

NDF - Binomial density function 

NDMAX - maximum number of points 
NDMIN - minimum number of points 

NEM - number of EM iterations 

NEXPO - negative exponential 

NIF - Binomial inverse function 

NIPALS - Nonlinear iterative partial least Squares 
NLAG - number of lags 

NLLOSS - nonlinear loss functions 
NLMODEL - nonlinear models 

NMIN - minimum count 

NMULTIVAR - multiple numeric variables 
NONLIN - nonlinear models 

NP-Number nonconforming 

NPAR - nonparametric 

NREC - non-recreationist 

NRN - Binomial random number 

NROW - number of rows 

NRP - number of apparently redundant 
parameters 

NSAMP - number of sub-samples 

NSPLIT - maximum number of splits 

NX - number of nodes along the x axis 
NXDIS - number of discretization points in the x 
(North) direction 

NY - number of nodes along the y axis 
NYDIS - number of discretization points in the y 
(East) direction 

NZ - number of nodes along the z axis 
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Acronyms 


NZDIS - number of discretization points in the z 
(Depth) direction 


(0) 

Obs-observed 

OBSFREQ - observed frequency 

OC - operating characteristic 

ODBC - open database capture and connectivity 
OFREQ - outlier frequencies 

OLS - ordinary least-squares 
ORTHEQ-Equally Spaced Orthogonal 
component 

ORTHUN- Unequally Spaced Orthogonal 
component 


P 

P - Proportion nonconforming 

PACF - Pareto cumulative function 
PACE - partial autocorrelation function 
PADF - Pareto density function 

PAIF - Pareto inverse function 
PARAM - parameters 

PARN - Pareto random number 

PCA - process capability analysis 
PCF - iterated principal axis factoring 
PCF - Poisson cumulative function 
PCNTCHANGE - percentage change 
PCT - Macintosh PICT 

PDF - Poisson density function 

pdf - probability density function 
PDL - polynomial distributed lag 
PERMAP - perceptual mapping 

PIF - Poisson inverse function 
PLIMITS - probability limits 

PLS - partial least squres 

pmf - probability mass function 
PMIN - minimum proportion 

PNG - Portable Network Graphics 
POLY - polygon 

POSAC - partially ordered scalogram analysis 
with coordinates 


P-P - probability plot 

PP - process performance 

Ppk - Process performance index for off-centered 

process 

PPL - process performance based on lower 

specification limit 

PPM - parts per million 

PPU - process performance based on upper 

specification limit 

PRE - percentage reduction error | 
PREFMAP - preference mapping | 
PRN - Poisson random number | 
PROB - probability 

PROP! - single proportion 

PROP2 - equality of two proportions | 
PS - PostScript | 
PVAF/p.v.a.f. -- present value annuity factor 

p-value - probability value 


Q 

QC - quality control 

QMLE - quasi maximum likelihood estimate 
QNTL - quantiles 

QPLOT - quantile plots 

Q-QPLOT - two sample quantile plot 
QRD - QR decomposition 

QS - quick search 

QSK - quantitative symmetric similarity 
coefficients (or Kulczynski measure) 
QUASI - Quasi-Newton method 


R 

R & R - repeatability and reproducibility 

R chart - range chart 

RADMAX - maximum horizontal direction for 
the search radius 

RADMIN - minimum horizontal direction for the 
search radius 

RAND - random 

RANDSAMP - random sampling 

RANKREG - rank regression 
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RBSTAT - row basic statistics 

RCF - Rayleigh cumulative function 

RDF - Rayleigh density function 
RDISCRIM - robust discriminant 

RDIST - robust distance 

RDVER - vertical direction for the search radius 
REPAR - reparametrize 

REPS - replicates 

RESID - residuals 

RIF - Rayleigh inverse function 

RJS - rejection sampling 

RMS - root mean square 

RMSEA - root mean square error of 
approximation 

RMSSTD - root mean square standard deviation 
ROC - receiver operating characteristic 
ROWPCT - Row percentages 

RRN - Rayleigh random number 

RS - response surface 

RSE- robust standard errors 

RSEED - random seed 

RSM- response surface methods 

RSQ - stress and squared correlation 

RSS - residual sum of squares 
RSTATISTICS - row statistics 

RTF - rich text format 

RWM-H - random walk Metropolis-Hastings 
RWSTEM - stem and leaf plot for rows 


S 

S chart - standard deviation control chart 

SANGI - angle (in degrees) of the first minor axis 
of the search ellipsoid 

SANG2 - angle (in degrees) of the major axis of 
the search ellipsoid 

SANG3 - angle (in degrees) of the second minor 
axis of the search ellipsoid 

SAV - SPSS files 

SB - smaller the better 

se - scale 

SC - set correlation 


Acronyms 


SCDFUNC - standardized coefficients for 
canonical variables 

SCF - Studentized cumulative function 
SD - standard deviations 

sd2/sas7bdat - SAS v9 files 

SDF - Studentized density function 
SE/se/S.E. - standard error 

SEK - standard error of kurtosis 

SEM - standard error of mean 

SES - standard error of skewness 

shp - shape 

SIF - Studentized inverse function 
SIMPLS - Straight-forward Implementation of 
Partial Least Squares 

SKMEAN - simple kriging mean 

SL - specification limit 

SMIN - minimum split value 

SPLOM - scatter plot matrix 

SQL - structured query language 
SQRT/SQR - square-root 

SRN - Studentized random number 
SRWR - sum of rank weighted residuals 
SS - sum of squares 

SSCP - sum of squares and cross products 
STA - Statistica v5 data files 

STAND - standardized deviates 

SVD - singular value decomposition 
SW - Shapiro-Wilks 

SYC/CMD - SYSTAT command Files 
SYZ/SYD/SYS - SYSTAT data files 
SYO - SYSTAT output files 


T 

T1 - one-sample t-test 

T2 - two-sample t-test 

TANALYZE - Taguchi design: analyze 
TCF - t cumulative function 

TCOR - total correlation 

TCOV - total covariance 

TDF -t density function 

TESTAT - Test Item Analysis 
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Acronyms 


TESTATCL - classical test item analysis 
TESTATLOG - logistic item response analysis 
TETRA - tetrachoric correlations 
TGENERATE - Taguchi design: generate 

TIF - t inverse function 

TIFF - Tagged Image File Format 

TLOG - log time 

TLOSS - Taguchi's Loss Function 

TNH - hyperbolic tangent 

TOHCO - Hypothesis Testing: Zero correlation 
TOHCI - Hypothesis Testing: Specific 
correlation 

TOHC2 - Hypothesis Testing: Equality of two 
correlation coefficients 

TOHPI - Hypothesis Testing: Single proportion 
TOHP2 - Hypothesis Testing: Equality of two 
proportions 

TOHTI - Hypothesis Testing: One sample t-test 
TOHT2 - Hypothesis Testing: Two sample t-test 
TOHTPAIRED - Hypothesis Testing: Paired t- 
test 

TOHVI - Hypothesis Testing: Single variance 
TOHV2 - Hypothesis Testing: Two variances 
TOHVN - Hypothesis Testing: Several variances 
TOHZI - Hypothesis Testing: One sample z-test 
TOHZ2 - Hypothesis Testing: Two sample z-test 
TOL - tolerance 

TPLOT - time series plot 

TPREDICT - Taguchi design: predict 

TRCF - Triangular cumulative function 

TRDF - Triangular density function 

TRI - triangular 

TRIF - Triangular inverse function 

TRIM - trimmed mean 

TRN - t random number 

TRP - transpose 

TRRN - Triangular random number 
TSFOURIER - Fourier decomposition of time 
Series 

TSIV - Two-Stage Instrumental Variables 

TSLS - Two-Stage Least Squares 


TSP - traveling salesman path 

TSQ chart - Hotelling's T? chart 
TSSMOOTH - smoothing time series 
TXT - text format 


U 

U chart - chart showing defects per unit 
UCF - Uniform cumulative function 
UCL - upper control limit 

UDF - Uniform density function 

UIF - Uniform inverse function 

UNCE - uncertainty coefficient 

URN - Uniform random number 

USL - upper specification limit 

UTL - upper tolerance limit 


V 
VAR - variance 
VIF - variance inflation factor 


WwW 

WB - Weibull 

WCF - Weibull cumulative function 
WCOR - pooled within-group correlation 
WCOV - pooled within-group covariance 
WDF - Weibull density function 
WHISKER - Box-and-Whisker plot 

WIF - Weibull inverse function 

WMF - Windows metafile 

WRN - Weibull random number 


X 

XCF - Chi-square cumulative function 
XDF - Chi-square density function 

XIF - Chi-square inverse function 

XLAG - separation distance between lags 
XLS - excel format 

XLTOL - tolerance for lags 

XMAX - maximum along x axis 

XMIN - minimum along x axis 
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X-MR chart - Individuals and moving range chart 
XPT/TPT - SAS transport files 

XRN - Chi-square random number 

XTAB - Crosstabulations 


¥ 
YMAX - maximum along y axis 
YMIN - minimum along y axis 


Z 

Z1 - one-sample z-test 

Z2 - two-sample z-test 

ZCF - Normal cumulative function 
ZDF - Normal density function 
ZICF - Zipf cumulative function 
ZIDF - Zipf density function 
ZIF - Normal inverse function 
ZIIF - Zipf inverse function 
ZIRN - Zipf random number 
ZMAX - maximum along z axis 
ZMIN - minimum along z axis 
ZRN - Normal random number 


Acronyms 


A 


A matrix, II-192 
accelerated failure time distribution, IV-433 
ACF plots, IV-529 
additive trees, I-80, 1-91 
AIC and Schwarz’s BIC, 11-39, 11-108, 11-292, Il- 
300, 11-344, 11-385, III-1, 111-258, IV-99, IV-427 
see linear models, 1-17 
Akaike Information Criterion, 111-458 
alpha level, IV-22, IV-28 
alternative hypothesis, 1-13, IV-20 
analysis of covariance, 11-153, 11-209 
examples, 11-170 
analysis of variance, II-107 
AIC and Schwarz's BIC, 11-108 
algorithms, 11-171 
assumptions, 11-25 
between-group differences, 11-32 
commands, 11-121 
compared to loglinear modeling, 11-95 
compared to regression trees, 1-45 
contrasts, 11-28, 11-113, 11-115, II-116 
data format, 11-121 
examples, 11-122, 11-126, 11-132, 11-145, I- 
146, 11-148, II-151, 1-155, 11-160, 
11-163, 11-166, 11-170 
factorial, 11-24 
homogeneity tests, 11-113 
hypothesis tests, 11-23, 11-113, II-115, II-116 
interactions, Il-25 
normality tests, II-112 
pairwise comparisons, 1-117 
power analysis, IV-19, IV-26, IV-55, IV-57, 
IV-77, IV-80 
Quick Graphs, II-121 
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repeated measures, 11-31, H-1 10 
resampling, II-108 
residuals, II-110 
sums of squares, 11-113 
two-way ANOVA, IV-26, IV-57, IV-80 
unbalanced designs, 11-29 
unequal variances, 11-26 
usage, II-121 
within-subject differences, 11-32 
Anderberg dichotomy coefficients, 1-164, 1-173 
Anderberg's binary similarity coefficient, 1-164 
Anderson-Darling test, 1-303 
Andrews procedure, 11-279 
angle tolerance, IV-388 
anisotropy, IV-392, IV-405 
geometric, IV-392 
zonal, IV-393 
A-optimality, 1-364 
ARIMA models, IV-514, IV-523, IV-540 
algorithms, IV-578 
arithmetic mean, 1-299, 1-308 
ARMA models, IV-519 
asymptotically distribution-free estimates, III-412 
autocorrelation plots, I-11, IV-516, IV-520 
Automatic Interaction Detection( AID), 1-45, 1-47 
autoregressive models, IV-516 
average run length curves, IV-134 
chart types, IV-137 
continuous distributions, IV-1 39 
discrete distributions, IV-140 
overview, IV-134 
probability limits, IV-137 
axial designs, 1-360 
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backward elimination, I-15 
bandwidth, IV-350, IV-355, IV-388 
optimal values, IV-356 
relationship with kernel function, IV-357 
basic statistics 
Anderson-Darling test, 1-303, 1-309 
columns, 1-307 
commands, 1-322 
Cronbach's alpha, 1-321 
examples, 1-324, 1-326, 1-327, 1-328, 1-333, I- 
338, 1-340, 1-341, 1-342 
geometric mean, 1-300, 1-308 
harmonic mean, 1-300, 1-308 
multivariate normality assessment, 1-303 
N-&P-tiles, 1-309 
overview, 1-297 
Quick Graphs, 1-323 
resampling, 1-298 
rows, 1-316 
Shapiro-Wilk test, 1-302, 1-309 
stem-and-leaf for columns, 1-314 
stem-and-leaf for rows, 1-320 
test for normality, 1-302 
trimmed mean, 1-299, 1-308 
usage, 1-323 
bayesian regression, II-50 
credibility intervals, I-50 
gamma prior, II-52 
normal prior, 11-52 
best linear unbiased estimates(BLUE), 11-344, II- 
386 
best linear unbiased predictors (BLUP), 11-344, TI- 
386 
beta level, IV-22 
between-group differences 
in analysis of variance, 11-32 
bias, II-15 
binary logit, III-2 
compared to multinomial logit, III-5 
binary trees, 1-43 
biplots, IV-6, IV-8 


bisquare procedure, III-279 

biweight kernel, IV-365 

Bonferroni inequality, 1-47 

Bonferroni test, 1-175, 11-27, II-118, 11-196, 11-307, 
11-394 

bootstrap, 1-19, 1-21 

box plot, 1-305 

Box-and-Whisker plots, IV-112 
Box-Behnken designs, 1-357, 1-380 
Box-Cox power transformation, IV-157 
Box-Hunter designs, 1-353, 1-373 
Bray-Curtis measure, 1-162, 1-172 
broad inference space, 11-280 
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c charts, IV-131 
C matrix, 11-193 
candidate sets 
for optimal designs, 1-363 
canonical correlation analysis 
data format, IV-304 
examples, IV-305, IV-308, IV-312 
interactions, IV-304 
model, IV-299 
nominal scales, IV-304 
overview, IV-291 
partialled variables, IV-300 
Quick Graphs, IV-305 
resampling, IV-291 
rotation, IV-303 
usage, IV-304 
canonical rotation, IV-7 
categorical data, III-321 
categorical predictors, 1-45 
Cauchy kernel, IV-365 
CCF plots, IV-531 
central composite designs, 1-356, 1-384 
centroid designs, 1-359 
CHAID, I-46, 1-47 
chi-square tests for independence, 1-229, 1-233, 1- 
242 
circle model 


in perceptual mapping, IV-5 
city-block distance, I-172, IIn-191 
classical analysis, IV-488 
classification and regression trees, 1-41 
classification functions, 1-396 
classification trees 

algorithms, 1-62 

basic tree model, I-42 

commands, I-54 

compared to discriminant analysis, 1-46, 1-49 

, 1-46 

data format, 1-54 

displays, 1-51 

examples, 1-55, 1-57, 1-59 

loss functions, 1-51 

missing data, 1-62 

mobiles, 1-41 

model, 1-51 

overview, 1-41 

pruning, 1-47 

Quick Graphs, 1-54 

resampling, 1-41 

saving files, 1-54 

stopping criteria, 1-47, 1-53 

usage, 1-54 
cluster analysis 

additive trees, I-91 

algorithms, 1-122 

clustering, 1-65 

commands, 1-93 

data types, 1-95 

distances, 1-84 

examples, 1-96, 1-105, 1-108, 1-109, I-111, I- 

112, I-115, I-116, 1-118, 1-120 

exclusive clusters, 1-66 

hierarchical clustering, 1-82 

k-means clustering, I-78 

k-medians clustering, I-79 

missing values, 1-122 

overlapping clusters, 1-66 

overview, 1-65 

Quick Graphs, I-95 

resampling, 1-66 
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saving files, 1-95 
usage, 1-95 
clustered data, 11-421 
clustering 
hierarchical clustering, 1-68 
k-clustering, 1-78 
validity, 1-87 
Cochran's test of linear trend, 1-234 
coefficient of alienation, III-190, 111-212 
coefficient of determination 
see multiple correlation 
coefficient of variation, 1-307 
Cohen's kappa, 1-226, 1-234 
communalities, 1-458 
compound symmetry, 11-32 
conditional logistic regression, [1-5 
confidence curves, 111-273 
confidence intervals, I-11, 1-307 
path analysis, 111-455 
conjoint analysis 
additive tables, 1-126 
algorithms, 1-152 
commands, 1-135 
compared to logistic regression, 1-132 
data format, 1-135 
examples, 1-136, 1-140, 1-143, 1-147 
missing data, 1-153 
model, 1-133 
multiplicative tables, 1-128 
overview, 1-125 
Quick Graphs, 1-135 
resampling, 1-125 
saving files, 1-135 
usage, 1-135 
constraints 
in mixture designs, 1-360 
contingency coefficient, 1-227 
contour plot, IV. -243 
contour plots, IV-401 
contrast coefficients, 11-31 
contrasts 
in analysis of variance, 11-28 
control charts 
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aggregated data, IV-120 
average run length curves, IV-136 
control limits, IV-121 
discrete control limits, IV-121 
operating characteristic curves, IV-135 
raw data, IV-120 
regression charts, IV-152 
sigma limits, IV-122 
convergence, III-98 
convex hulls, IV-398 
Cook's distance, II-12 
Cook-Weisberg graphical confidence curves, I- 
273 
coordinate exchange method, 1-363, 1-386 
correlations, 1-67, 1-157 
algorithms, 1-199 
binary data, 1-173 
canonical, IV-291 
commands, 1-177 
continuous data, 1-171 
data format, 1-178 
dissimilarity measures, 1-172 
distance measures, 1-172 
examples, I-179, 1-182, 1-185, 1-186, 1-188, I- 
192, 1-195, 1-196, 1-198 
missing values, 1-170, I-199, 111-135 
options, 1-174 
overview, 1-157 
power analysis, IV-19, IV-25, IV-42, IV-44 
Quick Graphs, 1-178 
rank-order data, 1-172 
resampling, 1-158 
saving files, I-179 
set, IV-291 
usage, 1-178 
correlograms, IV-403 
correspondence analysis, IV-2, IV-6 
algorithms, 1-218 
commands, 1-206 
data format, 1-206 
examples, 1-207, 1-214 
missing data, 1-218 
model, 1-204 


overview, 1-201 
Quick Graphs, 1-206 
resampling, 1-201 
simple correspondence analysis, 1-204 
usage, 1-206 
covariance matrix, 1-171, III-135 
covariance paths 
path analysis, 111-401 
covariograms, IV-387 
Cox-Snell residual plot, IV-434 
Cramer's V, 1-227 
critical level, 1-13 
Cronbach's alpha, IV-488, IV-489 
see basic statistics, 1-321 
crossover designs, 11-175 
crosstabulation 
commands, 1-244 
data format, 1-246 


examples, 1-248, 1-250, 1-253, 1-256, 1-257, I- 
258, 1-261, 1-263, 1-269, 1-271, I- 


273, 1-275, 1-277, 1-279, 1-293 

multiway, 1-237 

one-way, 1-220, 1-222, 1-228 

overview, 1-219 

Quick Graphs, 1-247 

resampling, 1-219 

standardizing tables, 1-221 

two-way, 1-220, 1-223, 1-231 

usage, 1-246 
cross-validation, 1-48, 1-396, 11-16, 111-360 
cumulative sum charts 

see cusum charts, IV-142 
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D matrix, 11-194, 11-288, 11-309, 11-355, 11-397 
D SUB-A (d,), IV-321 
dates, IV-430 
dendrograms, 1-65, 1-107 
dependence paths 
path analysis, 111-399 
descriptive statistics, 1-1 
see basic statistics, 1-307 


design of experiments, I-132, 1-368, 1-369 
axial designs, 1-360 
Box-Behnken designs, 1-357 
central composite designs, 1-356 
centroid designs, 1-359 
commands, 1-370 
examples, 1-371, 1-372, 1-373, 1-375, 1-377, I- 
379, 1-380, I-381, 1-382, 1-384, 1-386 
factorial designs, 1-349, 1-350 
lattice designs, 1-359 
mixture designs, 1-350, 1-357 
optimal designs, 1-350, 1-362 
overview, 1-345 
Quick Graphs, 1-371 
response surface designs, 1-350, 1-354 
screening designs, 1-360 
usage, 1-370 
determinant criterion 
see D-optimality 
Dice's binary similarity coefficient, I-164 
dichotomy coefficients, 1-164 
Anderberg, 1-173 
Jaccard, 1-173 
positive matching, 1-173 
simple matching, 1-173 
Tanimoto, 1-173 
difficulty, IV-507 
discrete choice model, II1-7 
compared to polytomous logit, III-8 
discrete gaussian convolution, IV-361 
discriminant analysis 
classical discriminant analysis, 1-400 
commands, 1-407 
data format, 1-408 
estimation, 1-401 
examples, 1-409, 1-413, 1-420, 1-427, 1-435, I- 
438, 1-444, 1-449 
linear discriminant function, 1-397 
model, 1-400 
multiple groups, 1-399 
options, 1-401 
overview, 1-391 
prior probabilities, 1-398 
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Quick Graphs, 1-408 
resampling, 1-391 
robust discriminant analysis, 1-399 
statistics, 1-404 
stepwise estimation, 1-401 
usage, 1-408 
discrimination parameter, IV-507 
dissimilarities 
direct, III-187 
indirect, III-187 
distance measures, 1-67, 1-157 
distances 
nearest-neighbor, IV-396 
distance-weighted least squares (DWLS) smoother, 
IV-361 
distributions 
Benford’s law, 1-499, 111-332, IV-86, IV-221 
beta, 1-500, 111-333, 111-335, IV-88, IV-222 
binomial, 1-499, 111-332, IV-86, IV-221 
Cauchy, 1-500, 111-333, 111-335, IV-88, IV-222 
chi-square, 1-500, 111-333, 111-335, IV-88, IV- 
222 
discrete uniform, 1-499, 111-332, IV-86, IV- 
221 
double exponential, 1-501, 111-335, IV-88, IV- 
222 
Erlang, 1-501, 111-335, IV-88, IV-222 
exponential, 1-501, 111-333, 111-336, IV-88, 
IV-222 
F, 111-333, 111-336, IV-88, IV-222 
gamma, I-501, 111-333, 111-336, IV-89, IV-222 
generalized lambda, IV-222 
geometric, 1-499, 111-332, IV-86, IV-221 
Gompertz, 1-501, 111-333, 111-336, IV-89, TV- 
222 
Gumbel, 1-501, 111-333, 111-336, IV-89, IV- 
222 
hypergeometric, 1-499, 111-332, IV-86, IV-221 
inverse Gaussian, 1-501, 111-333, 111-336, IV- 
89, IV-222 
logarithmic series, 1-499, 111-332, IV-87, IV- 
221 
logistic, 1-501, 111-333, 111-336, IV-89, IV-222 
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logit normal, 1-501, 111-333, 111-336, IV-89, 
IV-222 
loglogistic, I-501, 111-333, 111-336, IV-89, IV- 
222 
lognormal, 1-501, 111-333, 111-336, IV-89, IV- 
222 
negative binomial, 1-499, 111-333, IV-87, IV- 
221 
non-central chi-square, 111-333, 111-336, IV-89, 
IV-222 
non-central F, 111-333, 111-336, IV-89, IV-222 
non-central t, 111-333, 111-336, IV-89, IV-222 
normal, 1-501, 111-333, 111-336, IV-89, IV-222 
Pareto, 1-501, 111-333, 111-336, IV-89, IV-222 
Poisson, 1-499, 111-333, IV-87, IV-221 
Rayleigh, 1-501, 111-333, 111-336, IV-89, IV- 
223 
smallest extreme value, 1-501, 111-333, 111-336, 
IV-89, IV-223 
studentized maximum modulus, 111-333, III- 
336, IV-89 
Studentized range, 111-336 
studentized range, III-333, IV-89, IV-223 
t, 111-333, III-336, IV-89, IV-223 
triangular, 1-501, 111-334, 111-336, IV-89, IV- 
223 
uniform, 1-501, 111-334, 111-336, IV-89, IV- 
223 
Weibull, 1-501, 111-334, 111-336, IV-89, IV- 
223 
zipf, 1-499, 111-333, IV-87, IV-221 
dit plots, I-14 
D-optimality, I-364 
dot histogram plots, I-14 
Double, 111-333 
D-Prime (d' ), IV-320 
dummy codes, II-180 
Duncan test, 11-27, II-119, 11-197 
Dunnett test, 11-27, 11-119, 11-197 
Dunnett's T3 test, 11-27, 11-119, 11-197 
Dunn-Sidak test, 1-175 
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ECVI, 111-458 
edge effects, IV-398 
effect size 

in power analysis, IV-22, IV-23 
effects coding, II-20, II-180 | 
efficiency, 1-362 
eigenvalues, 1-405 
ellipse model 

in perceptual mapping, IV-6 
EM algorithm, 1-492 
EM estimation, III-130 

for correlations, 1-175, III-135 

for covariance, III-135 

for SSCP matrix, III-135 
endogenous variables 

path analysis, III-400 
Epanechnikov kernel, IV-364 
equamax rotation, 1-460, 1-464 
Erlang, 111-333 
Estimation, III-135 
Euclidean distances, III-188 
exogenous variables 

path analysis, 111-400 
expected cross-validation index, 111-458 
Exponential, 111-336 
exponential distribution, IV-432 
exponential model, IV-390, IV-404 
exponential smoothing, IV-524 
exponentially weighted moving average charts, IV- 
146 

control limits, IV-147 
external unfolding, IV-4 
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F, 111-333 
F and R matrices, 11-308, 11-354, 11-396 
F distribution 
F matrix, 11-287 
factor analysis, 1-457, IV-2 
algorithms, 1-492 
commands, 1-468 


compared to principal components analysis, I- 
460 
convergence, 1-463 
correlations vs covariances, 1-457 
eigenvalues, 1-463 
eigenvectors, 1-467 
examples, 1-469, 1-473, 1-476, 1-478, 1-482, l- 
485 
iterated principal axis, 1-463 
loadings, 1-467 
maximum likelihood, 1-463 
missing values, 1-492 
number of factors, 1-463 
overview, 1-453 
principal components, 1-463 
Quick Graphs, 1-468 
resampling, 1-453 
residuals, 1-465 
rotation, 1-459, 1-464 
save, 1-466 
scores, 1-466 
usage, 1-468 
factor loadings, IV-488 
factorial analysis of variance, 11-24 
factorial designs, 1-349, 1-350 
analysis of, 1-353 
examples, 1-371 
fractional factorials, 1-352 
full factorial designs, 1-352 
F-distribution 
non-centrality parameter, IV-60 
Fedorov method, 1-363 
Fieller bounds, 11-48 
filters, IV-527 
Fisher's exact test, 1-226, 1-233 
Fisher's linear discriminant function, IV-2 
Fisher's LSD, 11-197 
Fisher's LSD test, 11-27, 11-118, 11-307, 11-395 
fitting distributions 
commands, 1-501 
examples, 1-504, 1-505, 1-507, 1-508, 1-510, I- 
511, 1-513 
goodness-of-fit tests, 1-496 
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maximum likelihood method, 1-497 
method of moments, 1-497 
method of quantiles or order statistic, 1-497 
overview, 1-495 
Quick Graphs, 1-503 
Shapiro-Wilk's test for normality, 1-497 
usage, 1-503 
fixed effects, 11-279 
fixed variance 
path analysis, 111-402 
fixed-bandwidth method 
compared to KNN method, IV-357 
for smoothing, IV-355, IV-357, IV-364 
Fletcher-Powell minimization, IV-507 
forward selection, II-15 
Fourier analysis, IV-526, IV-545 
fractional factorial designs 
Box-Hunter designs, 1-353 
examples, 1-372, 1-373, 1-375, 1-377, 1-379 
homogeneous fractional designs, 1-353 
Latin square designs, 1-353 
mixed-level fractional designs, 1-353 
Plackett-Burman designs, 1-353 
Taguchi designs, 1-353 
Freeman-Tukey deviates, III-93, 11-102 
frequencies, 1-23, 1-54, 1-135, 1-179, 1-206, 1-246, 
1-248, 1-323, 1-408, 1-468, 1-469, 1-503, 1-544, Il- 
54, 11-121, 11-122, 11-202, 11-310, 11-357, 11-399, 
11-441, 111-23, 111-103, 111-104, 11-137, 111-194, MI- 
217, 111-283, 111-339, 111-364, 111-385, 111-413, IV- 
9, IV-62, IV-63, IV-103, IV-162, IV-244, IV-280, 
IV-305, IV-325, IV-328, IV-366, IV-410, IV-449, 
1V-495, IV-498, IV-547, IV-587 
frequency tables, 11-93, 11-102 
see crosstabulation 
Friedman test, 111-328 
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Gabriel test, 11-27, 11-119, 11-197 
Games-Howell test, 11-27, 11-119, 11-197 
Gaussian kernel, IV-364, IV-365 
Gaussian model, IV-390, IV-404 
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Gauss-Newton method, 111-269, 111-272 
general linear models, 11-175 
algorithms, 11-249 
categorical variables, II-179 
commands, 11-200 
contrasts, 11-189, 11-191 
data format, 11-201 
examples, 11-203, 11-211, 11-212, 11-213, I- 
215, 11-217, 11-220, 11-222, 11-224, 
11-234, 11-237, 11-238, 11-242, 11-246, 
11-247, 11-248 
hypothesis options, 11-188 
hypothesis tests, 11-186 
mixture model, 11-184 
model estimation, 11-177 
overview, 11-175 
pairwise comparisons, 11-195 
post hoc tests, I1-199 
Quick Graphs, 11-202 
resampling, 11-176 
stepwise regression, II-183 
usage, 11-201 
generalized least squares, 111-412, IV-584 
generalized variance, IV-294 
geometric mean, 1-300, 1-308 
geostatistical models, IV-386, IV-387 
getween-groups testing, 111-239 
Gini index, 1-48, 1-51 
GLM 
see general linear models, 11-175 
global criterion 
see G-optimality 
GMA chart, IV-146 
Goodman-Kruskal gamma, 1-227, 1-234 
Goodman-Kruskal lambda, 1-234 
goodness-of-fit tests, 1-496 
G-optimality, 1-364 
Gower2 binary similarity coefficient, 1-164 
Graeco-Latin square designs, 1-353 
Greenhouse-Geisser statistic, II-33 
Guttman mu2 monotonicity coefficients, 1-162 
Guttman's coefficient of alienation, III-190 
Guttman's loss function, III-212 


Guttman-Rulon coefficient, IV-489 
H 


Hadi outlier detection, 1-168 
Hamman's binary similarity coefficient, I-164 
Hampel procedure, 111-279 
Hanning weights, IV-512 
harmonic mean, 1-300, 1-308 
hazard function 
heterogeneity, IV-435 
Henderson's mixed model equations, 11-279, 11-293 
Henze-Zirkler test, I-303 
heteroskedasticity, IV-583 
heteroskedasticity-consistent standard errors, IV- 
583 
hierarchical clustering, 1-68, 1-82 
distances, 1-84 
validity index, I-75 
hierarchical linear mixed models 
categorical variables, 11-389 
commands, 11-398 
examples, 11-399, 11-402, 11-406, 11-408, Il- 
412, 11-414, 11-417 
hypothesis testing, 11-394 
model estimation, 11-387 
options, 11-392 
overview, 11-385 
Quick Graphs, 11-398 
random effects, 11-390 
usage, 11-398 
hierarchical linear models 
see mixed regression 
hinge, 1-301 
Hochberg's GT2 test, 11-27, 11-119, 11-197, 11-307, 
11-395 
hole model, IV-391, IV-405 
Holt's method, IV-524 
homogeneity tests, II-113 
Levene's test, 11-113 
Hotelling's T squared charts, IV-153 
Hotelling-Lawley trace, 111-226 
Huber procedure, 111-279 


Huynh-Feldt statistic, 11-33 
hyper-Graeco-Latin square designs, 1-353 
hypothesis 

alternative, I-13 

null, 1-13 

testing, I-12, II-7 
hypothesis testing 

Bartlett’s test, 1-521 

commands, 1-541 

confidence intervals, 1-520, 1-521, 1-522 

data format, 1-543 

examples, 1-544, 1-545, 1-547, 1-548, 1-549, I- 

551, 1-552, 1-556, 1-557, 1-560, I- 
562, 1-564 

Levene's tests, 1-521 

multiple tests, 1-522 

overview, 1-519 

Quick Graphs, 1-544 

resampling, 1-519 

test for means, 1-520 

tests for correlation, 1-522 

tests for mean, 1-520 

tests for proportion, 1-520, 1-538 

tests for variance, 1-521 

usage, 1-543 
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ID3, 1-47 
I-MR chart 

see X-MR chart, IV-150 
incomplete block designs, 1I-175 
independence, 1-223 

in loglinear models, 11-94 
individual cases charts 

See X charts, IV-129 
INDSCAL model, 111-185 
inertia, 1-202 
inferential statistics, 1-7, IV-20 
instrumental variables, IV-582 
intermediate inference space, 11-280 
internal-consistency, IV-489 
interquartile range, 1-301 
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interval censored data, IV-428 
inverse-distance smoother, IV-360 
isotropic, IV-387 
item-response analysis 

see test item analysis 
item-test correlations, IV-488 
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Jaccard dichotomy coefficients, 1-164, 1-173 
jackknife, I-18, 1-22 
jackknifed classification matrix, 1-396 


K 


k nearest-neighbors method 
compared to fixed-bandwidth method, IV-357 
for smoothing, IV-356, IV-362 
k-clustering, I-78 
k-means, I-78 
k-medians, 1-79 
Kendall’s Tau b, 1-172 
Kendall’s tau-b coefficient, 1-227 
kernel functions, IV-350, 1V-352 
biweight, IV-364 
Cauchy, IV-364 
Epanechnikov, IV-364 
Gaussian, IV-364 
plotting, IV-354 
relationship with bandwidth, IV-357 
tricube, IV-364 
triweight, IV-362, IV-364 
k-exchange method, 1-363 
Kolmogorov-Smirnov test, 111-319 


kriging, IV-405 
ordinary, IV-394, IV-405, IV-407 
simple, IV-393, IV-407 
trend components, IV-394 
universal, IV-394, IV-407 
Kruskal’s loss function, 11-211 
Kruskal’s STRESS, Ir-190 
Kruskal-Wallis test, 1-319 
K-S test, 111-319 
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Kulczynski's binary similarity coefficient, I-164 
Kulczynski's binary similarity coefficient, I-173 
kurtosis, 1-307 


L 


latent trait model, IV-488, IV-490 

Latin square designs, 1-353, 1-375 

lattice, III-382 

lattice designs, 1-359 

least absolute deviations, III-268 

least absolute deviations regression, IV-260 

least median of squares regression, IV-261 
search method, IV-269 

least trimmed squares regression, IV-261 

Levene test, II-25 

leverage, [I-12 

likelihood ratio chi-square, 1-233, 111-96, III-101 
compared to Pearson chi-square, III-96 

likelihood-ratio chi-square, 1-226 

Lilliefors test, 111-334, 111-355 

linear contrasts, 11-28 

linear discriminant model, 1-392 

linear mixed models 
categorical variables, 11-347 
commands, 11-356 


examples, 11-357, 11-362, 11-366, 11-369, II- 


372, 11-379, 11-382 
hypothesis testing, 11-352 
model estimation, 11-345 
options, 11-350 
Overview 
Quick Graphs, 11-356 
random effects, 11-348 
usage, 11-356 
linear models 
general linear models, 11-175 
hierarchical, 11-421 
linear discriminant model, 1-392 
linear regression, 11-39, 11-299, 11-385 
linear regression, 1-11, 11-7, 11-39 
AIC and Schwarz's BIC, 11-39 
Anderson-Darling test, 11-45 


bayesian, 11-50 
commands, 11-53 
data format, II-54 
examples, 11-55, 11-60, 11-63, 11-67, 11-71, H- 
75, 1-81, 11-83, 11-85, 11-86, 11-87, 
11-89, 11-95, 11-97, 11-99 
Kolmogorov-Smirnov test, 11-45 
model, 11-41 
normality tests, 11-45 
overview, 11-39 
prediction intervals, 11-40, 11-46 
Quick Graphs, 11-54 
resampling, 11-40, 11-47 
residuals, II-9, 11-41 
ridge, 11-48 
Shapiro-Wilk test, 11-45 
stepwise, 11-15 
tolerance, II-43 
usage, 11-54 
using correlation matrix as input, 11-18, 11-89 
using covariance matrix as input, 11-18, 11-89 
using SSCP matrix as input, 11-18, 11-89 
variance inflation factor, 11-70 
listwise deletion, 1-492, 111-125 
Little’s MCAR test, III-123, 111-133 
loadings, 1-456, 1-457 
LOESS smoothing, IV-361, IV-363, IV-367, IV- 
368, IV-370, IV-380 
logistic item-response analysis, IV-506 
one-parameter model, IV-490 
two-parameter model, IV-490 
logistic regression 
AIC and Schwarz’s BIC, III-1 
algorithms, III-85 
categorical predictors, III-11 
classification table, 11-17 
compared to conjoint analysis, 1-132 
conditional variables, III-10 
confidence intervals, [11-48 
data format, 111-22 
deciles of risk, 11-17 
discrete choice, 11-13 
dummy coding, III-11, 111-12 


effect coding, III-11, I-12 
estimation, III-15 
examples, III-24, III-27, III-33, 11-39, III-45, 
III-50, III-60, 11-69, 11-70, 11-77, 
I-81 
missing data, III-86 
model, HI-10 
options, III-14 
overview, III-1 
post hoc tests, I-20 
prediction table, III-16 
quantiles, III-18, III-49 
Quick Graphs, II1-23 
regression diagnostics, I-87 
robust standard errors, HI-16 
ROC curve, III-1 
simulation, III-19 
usage, III-22 
weights, III-23 
logit 
binary logit, III-2 
conditional logit, III-5 
discrete choice logit, II1-7 
multinomial logit, III-5 
stepwise logit, III-9 
loglinear modeling 
commands, III-103 
compared to analysis of variance, 11-95 
compared to Crosstabs, 11-102 
convergence, 11-96 
data format, III-103 
examples, III-105, III-114, 111-117, 11-121 
frequency tables, 111-102 
model, 11-96 
overview, 111-93 
parameters, III-100 
Quick Graphs, III-104 
saturated models, III-95 
statistics, III-100 
structural zeros, III-98 
usage, III-103 
log-logistic distribution, IV-432 
lognormal distribution, IV-432 


Index 


longitudinal data, 11-421 

loss function, 111-265 
multidimensional scaling, 111-210 

loss functions, 1-48 

LOWESS smoothing, IV-513 

low-pass filter, IV-527 

LSD test, 11-197 


M 


madograms, IV-403 
Mahalanobis distances, 1-392 
Mann-Whitney, 111-342 
Mantel-Haenszel test, 1-238 
Mardia skewness and kurtosis, 1-298, 1-303 
Marquardt method, 111-275 
Marron & Nolan canonical kernel width, IV-357, 
IV-364 
mass, 1-202 
matrix displays, 1-70 
maximum likelihood estimates, 11-385, 111-266 
maximum likelihood factor analysis, 1-461 
Maximum Wishart likelihood, 11-411 
McFadden’s conditional logit model, 11-7 
McNemar’s test, 1-226, 1-234 
MDPREF, IV-6, IV-8 
MDS 

see multidimensional scaling, 1-185 
mean, I-3, 1-307 
mean smoothing, IV-358, IV-365 
means coding, Il-21 
median, 1-4, 1-299, 1-307 
median smoothing, IV-358 
meta-analysis, I-19 
midrange, 1-301 
minimum spanning trees, IV-396 
Minkowski metric, III-191 
MIS function, 111-142 
Missing At Random(MAR), 11-131 
Missing Completely At Random(MCAR), 111-131 
missing value analysis 

casewise pattern table, 111-142 

data format, 111-137 


Index 


EM algorithm, 111-130, III-134, III-135, IHI- 
154, III-168, III-176 

examples, 111-137, III-142, 111-154, III-168, 
11-176 

listwise deletion, III-125, III-154, III-168 

MISSING command, 111-136 

missing value patterns, III-137 

model, III-134 

outliers, III-135 

overview, III-123 

pairwise deletion, 111-125, III-154, 111-168 

pattern variables, III-124, III-176 

Quick Graphs, III-137 

randomness, 111-131 

regression imputation, III-127, 111-134, MI- 
154, 111-176 

resampling, 111-123 

saving estimates, 111-134, 111-137 

unconditional mean imputation, 111-126 

usage, 111-137 


mixed models, 11-251 


AIC and Schwarz's BIC, 11-292 

ANOVA Method, II-281 

compound symmetry structure, 11-270 

covariance structures, II-269 

diagonal structure, 11-271 

estimation methods, 11-281 

hypothesis testing, 11-286 

MIVQUE(0) method, 11-283 

ML method, 11-284 

pairwise comparison, 11-290 

post hoc tests, 11-290 

REML method, II-285 

setup, 11-267 

unstructured (general symmetric structure), I- 
272 

variance components structure, 11-270 


mixed regression 


algorithms, 11-484 

commands, 11-441 

data format, 11-441 

examples, 11-442, 11-449, 11-457, 11-473 
overview, 11-421 


Quick Graphs, 11-441 
usage, 11-441 
mixture designs, 1-350, 1-357 
analysis of, 1-361 
axial designs, 1-360 
centroid designs, 1-359 
constraints, 1-360 
examples, 1-381, 1-382 
lattice designs, 1-359 
Scheffé model, 1-361 
screening designs, 1-360 
simplex, 1-359 
models, 1-10, 11-301 
estimation, I-10 
moving average, IV-355, IV-511, IV-517 
moving average chart, IV-144 
moving-averages smoother, IV-360 
M-regression, IV-261 
multidimensional scaling, III-185, IV-2 
algorithms, III-211 
assumptions, III-186 
commands, 111-194 
configuration, III-189, 111-193 
confirmatory, 111-193 
convergence, 111-192 
data format, III-194 
dissimilarities, 11-187 
distance metric, 111-189 
examples, III-195, III-198, 111-200, 111-203, 
111-208 
Guttman method, 111-212 
individual differences, 111-185 
Kruskal method, 111-211 
log function, III-191 
loss function, III-190 
metric, 111-189 
missing values, 111-212 
nonmetric, 111-189 
overview, III-185 
power function, III-191 
Quick Graphs, 111-194 
residuals, III-192 
R-metric, III-191 
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Shepard diagrams, III-189, 111-194 
usage, III-194 
multilevel models 
see mixed regression 
multinomial logit, III-5 
compared to binary logit, III-5 
multinormal tests, III-215 
examples, 111-218, [1-219 
Henze-Zirkler test, III-215 
Mardia skewness and kurtosis, III-215 
overview, 111-215 
Quick Graphs, 11-217 
usage, 111-217 
using commands, 111-217 
multiple comparison tests 
see pairwise comparisons, II-1 17, 11-195 
multiple correlation, [1-8 
multiple correspondence analysis, 1-203 
multiple regression, I-12 
multiple tests 
Bonferroni adjustment, 1-522 
Dunn-Sidak adjustemnt, 1-522 
multivariate analysis of variance, 111-223 
between-groups testing, 111-239 
categorical variables, 111-229 
commands, 111-244 
data format, 111-244 
examples, 111-246, 111-248, 111-253, 111-255, 
111-257, 111-258 
Hotelling-Lawley trace, 111-226 
hypothesis test, 111-232 
overview, 111-223 
Pillai trace, 11-225 
post hoc test, 111-242 
Quick Graphs, 111-245 
repeated measures, 111-230 
Roy’s Greatest root, 11-226 
usage, 111-244 
Wilks’ lambda, 111-225 
within-group testing, III-241 
multivariate normality assessment 
Henze-Zirkler test, 1-303 
Mardia's skewness, 1-303 


Index 


mutually exclusive, 1-222 


N 


N- & P-tiles, 1-309 
methods, 1-311 
transformation, 1-309 
Nadaraya- Watson smoother, IV-360 
narrow inference space, 11-280 
Nelson-Aalen cumulative hazard estimator, IV-438 
nesting, II-175 
Newton-Raphson method, I-93 
NIPALS (Nonlinear Iterative PArtial Least Squares) 
see partial least squares regression, 111-377 
nodes, 1-43 
nominal data, 111-321 
non-central F-distribution, IV-34, IV-60 
non-centrality parameters, IV-34 
nonlinear models, III-261 
algorithms, 11-316 
commands, 11-283 
computation, 111-274, 11-316 
convergence, 111-274, 11-275 
data format, 111-283 
estimation, 11-269 
examples, 111-284, 111-287, 111-290, 111-293, 
111-296, 111-298, 111-299, 111-301, MI- 
306, 111-311, 111-313, 11-315 
functions of parameters, 11-277 
loss functions, III-265, 111-270, 11-280, II- 
281 
missing data, 111-316 
model, 111-270 
ter bounds, 111-274 
problems, 111-269 
Quick Graphs, 111-283 
recalculation of parameters, 111-276 
resampling, III-261 
robust estimation, 111-278 
starting values, 111-274 
usage, 111-283 
nonmetric unfolding model, TH-185 
nonparametric statistics, 111-325 


Index 


nonparametric tests 
algorithms, III-355 
Anderson-Darling test, 111-334 
commands, 111-325, 111-331, 111-338 
data format, 111-339 
examples, 111-340, 111-342, 111-343, 111-345, 
111-346, 111-347, 111-348, 111-349, III- 
350, 111-353, 111-354 
Friedman test, 111-328 
independent samples test, 111-322, 111-323 
Kolmogorov-Smirnov test, 111-323, 111-331 
Kruskal-Wallis test, 111-322 
Mann-Whitney test, 111-322 
overview, 111-319 
Quade test, 111-329 
Quick Graphs, 111-339 
related variables tests, 111-325, 111-326, 111-328 
resampling, 111-319 
sign test, 111-325, 111-326 
usage, 111-339 
Wald-Wolfowitz runs test, 111-337 
Wilcoxon Signed-Rank test, 111-326 
normal distribution, 1-301 
normality tests, 11-45, 11-112 
Anderson-Darling, II-113 
Anderson-Darling test, 11-45 
Kolmogorov-Smirnov test, II-45, II-112 
Shapiro-Wilk, II-112 
Shapiro-Wilk test, 11-45 
np charts, IV-129 
NPAR, IV-320 
null hypothesis, I-12, IV-20 
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oblimin rotation, 1-460, 1-464 
observational studies, 1-347 

OC curves, IV-134 

Occam's razor, 1-130 

Ochiai's binary similarity coefficient, 1-164 
odds ratio, 1-233 

omni-directional variograms, IV-388 
operating characteristic curves 


chart type, IV-136 
continuous distributions, IV-139 
discrete distributions, IV-140. 
overview, IV-134 
probability limits, IV-136 
sample size, IV-138 
scaling, IV-138 

optimal designs, 1-350, 1-362 
analysis of, 1-364 
A-optimality, 1-364 
candidate sets, 1-363 
coordinate exchange method, 1-363, 1-386 
D-optimality, 1-364 
efficiency criteria, I-364 
Fedorov method, 1-363 
G-optimality, 1-364 
k-exchange method, 1-363 
model, 1-365 
optimality criteria, 1-364 

optimality, 1-362 

ORDER, IV-431 

ordinal data, 111-320 

Ordinary least squares, 111-412 

orthomax rotation, 1-460, 1-464 

Output, IV-99 


P 


p charts, IV-130 

PACF plots, IV-530 

pairwise comparisons, II-26, 11-107, 11-117 
Bonferroni test, 11-118, II-196 
Duncan test, 11-119, II-197 
Dunnett test, 11-119, 11-197 
Dunnett's T3 test, 11-119, 11-197 
Fisher's LSD, 11-197 
Fisher's LSD test, 11-118 
Gabriel test, II-119, 11-197 
Games - Howell test, 11-197 
Games-Howell test, 11-119 
Hochberg's GT2 test, 11-119 
Hochberg's test GT2, 11-197 
R-E-G-W Q test, 11-197 


R-E-G-W-Q test, 11-119 
Scheffé test, 11-27, 11-118, 11-197 
Sidak test, I1-118, 11-197 
Student-Newman-Keuls test, 11-119, 11-197 
Tamhane’s T2 test, 11-119, 11-197 
Tukey test, 11-118, 1-196 
Tukey’s b test, 11-119, 1-197 
pairwise deletion, 1-492, III-125 
parameters, 1-10 
parametric modeling, IV-432 
Pareto charts, IV-111 
partial autocorrelation plots, IV-519, IV-520 
partial least squares regression 
algorithms, 111-377 
cross-validation, III-363 
examples, 111-365, 111-368, 11-371, 11-375 
latent factors, 111-357, 111-359 
leave-one-out, 111-360, 111-363 
NIPALS, 111-362 
PRESS statistic, 111-360 
Quick Graphs, 111-364 
random exclusion, 111-360, 111-364 
SIMPLS, 111-362 
test set, 111-360 
training set, 111-360 
usage, 111-364 
using commands, 111-364 
partialing 
in set correlation, IV-295 
partially ordered scalogram analysis with coordi- 
nates 
algorithms, 111-395 
commands, 111-385 
Convergence, 111-384 
convergence, 111-384 
data format, 111-385 
displays, 111-383 
examples, 111-386, 111-388, 111-390 
missing data, 111-395 
model, 111-384 
overview, 111-381 
Quick Graphs, 111-385 
resampling, 11-381 


1 ndex 


usage, 111-385 


path analysis 


algorithms, 111-454 

confidence intervals, 111-455 
covariance paths, 111-401 
covariance relationship, 111-409 
data format, 111-413 
dependence paths, 111-399 
dependence relationship, 111-407 
endogenous variables, 111-400 
estimate, 111-411 

examples, 111-414, 111-419, 111-434, 111-442 
exogenous variables, 111-400 
fixed variance, 111-402 

free parameters, 11-418 

latent variables, III-404 
manifest variables, II1-410 
measures of fit, [11-455 

method of estimation, 11-41 1 
model, 111-452 

model statement, 111-407 
options, III-411 

overview, 111-397 

path diagrams, 111-397 

Quick Graphs, 111-413 

starting values, III-412 

usage, 111-413 

variance paths, III-401 


Pearson chi-square, 1-223, 1-228, 1-233, I-94, II- 


101 


compared to likelihood ratio chi-square, 11-96 


Pearson correlation, 1-160, 1-171 
perceptual mapping 


algorithms, IV-16 

commands, IV-9 

data format, IV-9 

examples, IV-9, IV-11, IV-12, IV-14 
methods, IV-8 

missing data, IV-16 

model, IV-7 

overview, IV-1 

PREFMAP, IV-1 

Quick Graphs, IV-9 
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Index 


usage, IV-9 
periodograms, IV-527 
permutation tests, 1-222 
phi coefficient, 1-48, 1-51, 1-52, 1-227 
Pillai trace, III-225 
Plackett-Burman designs, 1-353, 1-379 
point processes, IV-386, IV-395 
polynomial contrasts, 11-28, 11-31, II-192 
polynomial smoothing, IV-358, IV-365 
populations, 1-7 
POSET, 111-381 
positive matching dichotomy coefficients, I-164, I- 
173 
Post hoc Test for Repeated measures, 111-242 
power, IV-22 
power analysis 
analysis of variance, IV-19 
commands, IV-62 
correlation coefficients, IV-25, IV-42, IV-44 
correlations, IV-19 
data format, IV-62 
examples, IV-63, IV-67, IV-72, IV-77, IV-80 
generic, IV-34, IV-60, IV-77 
one-sample t-test, IV-26 
one-sample z-test, IV-46 
one-way ANOVA, IV-26, IV-55, IV-77 
overview, IV-19 
paired t-test, IV-26, IV-51, IV-67 
power curves, IV-62 
proportions, IV-19, IV-25, IV-39, IV-40, IV- 
63 
Quick Graphs, IV-62 
randomized block designs, IV-19 
t-tests, IV-19 
two-sample t-test, IV-53, IV-72 
two-sample z-test, IV-48 
two-way ANOVA, IV-26, IV-57, IV-80 
usage, IV-62 
z-tests, IV-19 
power curves, IV-62 
overlaying curves, IV-67 
response surfaces, IV-67 
Power model, IV-391, IV-405 


prediction intervals, 11-40, 11-46 
preference curves, IV-4 
preference mapping, IV-2 
PREFMAP, IV-7 
PRESS statistic 
in partial least squares regression, III-360 
principal components, 1-463 
principal components analysis 
coefficents, 1-456 
compared to factor analysis, 1-460 
compared to linear regression, 1-455 
loadings, 1-456 
prior probabilities, 1-398 
probability calculator 
examples, IV-90, IV-93, IV-94, IV-95 
overview, IV-85 
usage, IV-90 
probability limits, IV-121 
probability plots, 1-15, 11-9 
probit analysis 
AIC and Schwarz's BIC, IV-99 
algorithms, IV-107 
categorical variables, IV-102 
commands, IV-103 
data format, IV-103 
dummy coding, IV-102 
effect coding, IV-103 
examples, IV-104, IV-106 
interpretation, IV-100 
missing data, IV-107 
model, IV-100 
overview, IV-99 
Quick Graphs, IV-103 
saving files, IV-103 
usage, IV-103 
process capability analysis, IV-155 
Box-Cox power transformation, IV-157 
non-normal data, IV-157, IV-158 
process performance, IV-158 
Procrustes rotations, IV-7 
proportional hazards models, IV-433 
proportions 
power analysis, IV-19, IV-25, IV-39, IV-40, 


17 


IV-63 
p-value, IV-20 
Q 
QSK 


coefficients, 1-172 
Quade test, III-329 
multiple comparisons, 111-329 
pairwise comparisons, 111-330 
quadrat counts, IV-385, IV-398 
quadratic contrasts, 11-28 
quality analysis, IV-109 
aggregated data, IV-120 
average run length curves, IV-136 
Box-and-Whisker plots, IV-112 
commands, IV-161 
control charts, IV-114 
control limits, IV-121 
cusum charts, IV-142 
data format, IV-162 
discrete control limits, IV-121 
examples, IV-163, IV-164, IV-165, IV-166, 
IV-167, IV-168, IV-176, IV-178, IV- 
180, IV-183, IV-189, IV-191, IV- 
195, IV-197, IV-198, IV-199, IV- 
201, IV-203, IV-204, IV-206, IV- 
207, 1V-209, IV-212, IV-213, IV- 
215 
histogram, IV-110 
moving average chart, IV-144 
moving range, IV-149 
operating characteristic curves, IV-135 
overview, IV-109 
Pareto charts, IV-111 
process capability analysis, IV-155 
quick graphs, IV-162 
raw data, IV-120 
regression charts, IV-152 
run charts, IV-114 
run tests, IV-118 
shewhart control charts, IV-116 
sigma limits, IV-122 
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TSQ charts, IV-153 

usage, IV-162 

X-MR charts, IV-149 
quantile plots, IV-434 
quantitative symmetric dissimilarity coefficient, I- 
162 
quartimax rotation, 1-460, 1-464 
quasi-independence, 111-98 
Quasi-Newton method, 111-269, 111-273 


R 


R charts, IV-128 

R charts:plotting with X-bar charts, IV-129 

R matrix, 11-289 

Ramsay procedure, 111-279 

random coefficient models 
see mixed regression 

random effects, 11-259, 11-390 
in mixed regression, 11-421 

random fields, IV-386 

random samples, 1-8 

random sampling 
algorithms, IV-228 
commands, IV-223 
examples, IV-225, IV-226 
overview 
Quick Graphs, IV-224 
univariate continuous, IV-222 
univariate discrete, IV-220 
usage, IV-224 

random variables, Il-6 

random walk, IV-517 

randomized block designs, IV-37 
power analysis, IV-19 

range, 1-301, 1-307, IV-392 

Rank, IV-262 

rank regression, IV-262 

rank-order coefficients, 1-172 

Rasch model, IV-490 

receiver operating characteristic curves 
See signal detection analysis 


regression 
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bayesian regression, II-50 

LAD regression, IV-260 

Least-squares regression, IV-256 

linear, I-11 

LMS regression, IV-261 

logistic, III-1 

LTS regression, IV-261 

M-regression, IV-261 

rank regression, IV-262 

ridge regression, II-48 

S regression, IV-262 

TSLS regression, IV-581 

two-stage least squares, IV-581 
regression charts, IV-152 
regression trees, 1-45 

algorithms, 1-62 

basic tree model, 1-42 

commands, 1-54 


compared to analysis of variance, I-45 
compared to stepwise regression, I-46 


data format, 1-54 
displays, I-51 
examples, 1-55, 1-57, 1-59 
loss functions, 1-48, 1-51 
missing data, 1-62 
mobiles, 1-4] 
model, 1-51 
overview, 1-41 
pruning, 1-47 
Quick Graphs, 1-54 
resampling, 1-41 
saving files, 1-54 
stopping criteria, 1-47, 1-53 
usage, 1-54 
R-E-G-W Q test, 11-197 
R-E-G-W-Q test, 11-27, 11-119 
reliabilities, IV-492 
reliability, IV-489 
repeated measures, II-3] 
assumptions, 11-32 
resampling 
algorithms, 1-38 
bootstrap-t method, I-19 


command, I-22 


examples, 1-23, I-27, I-28, I-33, 1-34, 1-36 


missing data, 1-38 
naive bootstrap, I-19 
overview, I-17 
Quick Graphs, I-22 
usage, I-22 
response optimization, IV-234 
canonical analysis, IV-234 
desirability analysis, IV-236 
ridge analysis, IV-235 
response surface designs, 1-350, 1-354 
analysis of, 1-357 
Box-Behnken designs, 1-357 
central composite designs, I-356 
examples, 1-380, 1-384 
rotatability, 1-355, 1-356 
response surface methods, IV-231 
commands, IV-244 


contour and surface plot, IV-233, IV-243 


customization, IV-238 
estimate model, IV-237, IV-238 


examples, IV-245, IV-247, IV-249, IV-250 


lack of fit, IV-233 
optimize, IV-240 
overview, IV-231 
Quick Graphs, IV-244 
usage, IV-244 

response surfaces, 1-132, 111-273 


restricted/residual maximum likelihood estimates 


11-385 
ridge regression, 11-48 
right censored data, IV-428 
RMSEA, III-457 
robust discriminant analysis, 1-399 
robust regression 

commands, IV-279 


examples, IV-280, IV-283, IV-284 


LAD regression, IV-260 
LMS regression, IV-261 
LTS regression, IV-26] 
M-regression, IV-26] 
overview, [V-255 


Quick Graphs, IV-279 

rank regression, IV-262 

S regression, IV-262 

usage, IV-279 
robust smoothing, IV-358, IV-365 
robustness, 11-321 
ROC curves, IV-320 
root mean square error of approximation, 111-457 
rotatability 

in response surface designs, 1-355 
rotatable designs 

in response surface designs, 1-356 
rotation, 1-459 
Roy's Greatest root, III-226 
running median smoothers, IV-512 
running-means smoother, IV-360 


S 


8 charts, IV-126 
plotting with X-bar charts, IV-129 
Sakitt D, IV-321 
sample size, IV-23, IV-30 
samples, I-8 
saturated models 
loglinear modeling, III-95 
scale regression, IV-262 
scalogram 
see partially ordered scalogram analysis with 
coordinates 
scatterplot matrix, 1-160 
Scheffé model 
in mixture designs, 1-361 
Scheffé test, 11-27, 11-118, 11-197, 11-307, 11-395 
screening designs, 1-360 
SD-RATIO, IV-321 
seasonal decomposition, IV-523 
second-order stationarity, IV-387 
semi-variograms, IV-388 
set correlations 
assumptions, IV-292 
categorical variables, IV-301 
data format, IV-304 
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measures of association, IV-293 
missing data, IV-316 
overview, IV-291 
partialing, IV-292 
usage, IV-304 
Shapiro-Wilk test, 1-302 
Shepard diagrams, III-189, 111-194 
Shepard's smoother, IV-360 
Shewhart control charts 
e charts, IV-131 
np charts, IV-129 
p charts, IV-130 
R charts, IV-128 
s charts, IV-126 
u charts, IV-133 
variance charts, IV-124 
X charts, IV-129 
X-bar charts, IV-123 
Sidak test, 11-27, II-118, 11-197, 11-307, 11-395 
sign test, 111-325, 111-326 
signal detection analysis 
algorithms, IV-346 
chi-square model, IV-323 
commands, IV-324 
convergence, IV-324 
data format, IV-325 
examples, IV-328, IV-333, IV-335, IV-336, 
IV-340, IV-342, IV-344 
exponential model, IV-323 
gamma model, IV-323 
logistic model, IV-323 
missing data, IV-346 
nonparametric model, IV-323 
normal model, IV-323 
overview, IV-319 
poisson model, IV-323 
Quick Graphs, IV-327 
ROC curves, IV-327 
usage, IV-325 
sill, IV-392 
similarity measures, 1-157 
simple matching dichotomy coefficients, 1-164, I- 
173 
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simplex, 1-359 
Simplex method, 111-269, 111-273 
SIMPLS (Straight-forward IMplementation of Par- 
tial Least Squares) 
see partial least squares regression 
, 11-377 
simulation, IV-394 
singular value decomposition, 1-201, IV-6, IV-16 
skewness, 1-307 
positive, I-4 
slope, II-13 
smoothing, IV-362, IV-510 
bandwidth, IV-350, IV-355 
biweight kernel, IV-362, IV- 364, IV-365 
Cauchy kernel, IV-362, IV-365 
commands, IV-366 
confidence intervals, IV-368 
data format, IV-366 
discontinuities, IV-360 
discrete gaussian convolution, IV-361 
distance-weighted least squares (DWLS), IV- 
361 
Epanechnikov kernel, IV-362, IV-364 
examples, IV-367, IV-368, IV-370, IV-380 
fixed-bandwidth method, IV-355, IV-362, IV- 
364 
Gaussian kernel, IV-362, IV- 364, IV-365 
grid points, IV-361, IV-362, IV-382 
inverse-distance, IV-360 
k nearest-neighbors method, IV-356 
kernel functions, TV- 350, IV-352, IV-362, IV- 
364 
LOESS smoothing, IV-361, IV-362, IV-367, 
IV-368, IV-370, IV-380 
Marron & Nolan canonical kernel width, IV- 
357, IV-362, IV-364 
mean smoothing, IV-358, IV-365 
median smoothing, IV-358 
methods, IV-350, IV-358, IV-365 
model, IV-362 
moving-averages, [V-360 
Nadaraya-Watson, IV-360 
nonparametric vs. parametric, IV-350 


overview, IV-349 

polynomial smoothing, IV-358, IV-365 

Quick Graphs, IV-366 

resampling, IV-349 

residuals, IV-362, IV-366 

robust smoothing, IV-358, IV-365 

running-means, IV-360 

saving results, IV-364, IV-366, IV-367 

Shepard's smoother, IV-360 

step, IV-361 

tied values, IV-361 

tricube kernel, IV-364, IV-365 

trimmed mean smoothing, IV-365 

triweight kernel, IV-364, IV-365 

uniform kernel, IV-364 

usage, IV-366 

window normalization, IV- 357, IV-364 
Sneath and Sokal's binary similarity coefficient, 1- 
164 
Somers’ d coefficients, 1-227, 1-235 
Sorting, 1-5 
spaghetti plot, II-458 
spatial statistics, IV-385 

algorithms, IV-426 

azimuth, IV-403 

commands, IV-408 

data, IV-410 

dip, IV-403 

examples, IV-411, IV-417, IV-418, IV-424 

grid, IV-407 

kriging, IV-393, IV-400, IV-405 

lags, IV-402 

missing data, IV-426 

model, IV-385, IV-403 

nested models, IV-392 

nesting structures, IV-403 

nugget, IV-392 

nugget effect, IV-392, IV-405 

plots, IV-401 

point statistics, IV-400 

Quick Graphs, IV-410 

resampling, IV-385 

sill, IV-405 
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simulation, IV-394, IV-401 

spherical model, IV-404 

trends, IV-406 

usage, IV-410 

variogram, IV-400 
Spearman coefficients, I-162, I-172, 1-227 
Spearman-Brown coefficient, IV-489 
specificities, I-458 
spectral models, IV-510 
spherical model, IV-389 
split plot designs, II-175 
split-half reliabilities, IV-492 
SSCP matrix, III-135 
standard deviation, 1-3, 1-301, 1-307 
standard error of estimate, II-7 
standard error of skewness, 1-307 
standard error of the mean, 1-11, 1-307 
standardization, 1-67 
standardized alpha, IV-489 
standardized deviates, 1-202 
standardized values, 1-6 
stationarity, IV-387, IV-520 
statistics 

defined, I-1 

descriptive, I-1 

inferential, I-7 
stem-and-leaf plots, 1-3, 1-299 
step smoother, IV-361 
stepwise regression, 11-15, II-30, III-9 
stochastic processes, IV-386 
stress, III-188, III-211 
structural equation models 

see path analysis 
Stuart's tau-c coefficients, I-227, 1-234 
Student, 11-197 
studentized residuals, II-10 
Student-Newman-Keuls test, II-27, 11-119 
subpopulations, 1-305 
subsampling, I-18 
sum of cross-products matrix, I-171 
sums of squares 

type I, 11-29, 11-34, 11-113 

type II, 11-35, 11-113 
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type III, 11-30, 11-36, 11-113 
type IV, 11-36 


surface plot, IV-243 
surface plots, IV-401 
survival analysis 


AIC and Schwarz's BIC, IV-427 

algorithms, IV-476 

censoring, IV-428, IV-435, IV-479 

centering, IV-477 

coding variables, IV-437 

commands, IV-447 

convergence, IV-481 

Cox regression, IV-441 

data format, IV-448 

estimation, IV-442 

examples, IV-449, IV-453, IV-455, IV-459, 
IV-462, IV-464, IV-468, IV-472 

exponential model, IV-441 

graphs, IV-437, IV-444 

logistic model, IV-441 

log-likelihood, IV-477 

lognormal model, IV-435, IV-477 

missing data, IV-476 

model, IV-435 

models, IV-479 

Nelson-Aalen cumulative hazard estimator, IV- 
438 

overview, IV-427 

parameters, IV-476 

plots, IV-481 

proportional hazards models, IV-479 

Quick Graphs, IV-448 

Singular Hessian, IV-478 

stepwise, IV-482 

stepwise estimation, IV-443 

tables, IV-437, IV-444 

time dependent covariates, IV-446 

usage, IV-448 

variances, IV-483 

weibull model, IV-472 


symmetric matrix, 1-160 
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t tests 
Taguchi designs, 1-353, 1-377 
Tamhane’s T2 test, 11-27, II-119, 11-197 
Tanimoto dichotomy coefficients, I-164, 1-173 
tau-b coefficients, 1-234 
tau-c coefficients, 1-234 
test for normality, 1-302 
Anderson-Darling test, 1-303 
Shapiro-Wilk test, 1-302 
test item analysis 
algorithms, IV-506 
classical analysis, IV-488, IV-489, IV-491, 
IV-506 
commands, IV-494 
data format, IV-495 
examples, IV-498, IV-500, IV-503 
logistic item-response analysis, IV-490, IV- 
493, IV-506 
missing data, IV-507 
overview, IV-487 
Quick Graphs, IV-497 
reliabilities, IV-492 
resampling, IV-487 
scoring items, IV-492, IV-493 
statistics, IV-495 
usage, IV-495 
tests for correlation, 1-535 
equality of two correlations, 1-522, 1-537 
specific correlation, I-522, 1-536 
zero correlation, 1-522, 1-535 
tests for mean, 1-523 
one-sample t, 1-520, 1-526 
one-sample z, 1-520, 1-523 
paired t, 1-521, 1-527 
poisson, 1-520, 1-530 
two-sample t, I-521, 1-528 
two-sample z, 1-520, 1-524 
tests for normality 
AD test, 111-334 
K-S test, 111-331 
Lilliefors test, 111-334 


Shapiro-Wilk’s test, 1-497 
tests for proportion, 1-538 
equality of proportions, 1-521 
equality of two proportions, 1-540 
single proportion, I-520, 1-538 
tests for variance, 1-531 
Bartlett's test, 1-521 
equality of several variances, 1-534 
equality of two variances, I-521, 1-532 
Levene's test, 1-521 
single variance, 1-531 
tetrachoric correlation, 1-164, 1-166 
theory of signal detectability (TSD), IV-319 
time domain models, IV-510 
time series, IV-509 
algorithms, IV-578 
ARIMA models, IV-514, IV-540 
clear series, IV-534 
commands, IV-532, IV-534, IV-539, IV-540, 
IV-542, IV-544, IV-546 
data format, IV-546 
examples, IV-547, IV-548, IV-549, IV-550, 
IV-552, IV-555, IV-557, IV-558, IV- 
560, IV-561, IV-566, IV-575 
forecasts, IV-538 
Fourier transformations, IV-545 
missing values, IV-509 
moving average, IV-511, IV-535 
overview, IV-509 
plot labels, IV-528 
plots, IV-528, IV-529, IV-530, IV-531 
Quick Graphs, IV-546 
running means, IV-512, IV-535 
running medians, IV-512, IV-536 
seasonal adjustments, IV-523, IV-539 
smoothing, IV-510, IV-535, IV-536, IV-537 
stationarity, IV-520 
transformations, IV-532, IV-534 
trend analysis, IV-525, [V.542 
trends, IV-538 
usage, IV-546 
tolerance, II-16 
T-plots, IV-529 
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trace criterion 
see A-optimality 
tree clustering methods, I-47 
tree diagrams, I-70 
trend analysis, IV-525, IV-542 
Homogeneity test, IV-544 
Mann-Kendall test, IV-526, IV-543 
Modified Seasonal Kendall test, IV-543 
Seasonal Kendall test, IV-526, IV-543 
slope estimator, IV-573 
triangle inequality, 111-186 
tricube kernel, IV-364 
trimmed mean, 1-299, 1-308 
trimmed mean smoothing, IV-365 
triweight kernel, IV-364 
t-tests, IV-19 
one-sample, 1-526, IV-50 
paired, 1-527, IV-51 
power analysis, IV-26 
two-sample, 1-528, IV-53 
Tukey procedure, 111-279 
Tukey test, 11-27, 11-118, 11-196 
Tukey’s b test, 11-27, 11-119, 11-197 
Tukey’s HSD test, 11-307, 11-395 
Tukey’s jackknife, 1-18 
twoing, 1-48 
two-stage least squares 
algorithms, IV-597 
commands, IV-586 
estimation, IV-582 
examples, IV-587, IV-590, IV-592, IV-593, 
IV-595, IV-596 
heteroskedasticity-consistent standard errors, 
IV-586 
lagged variables, IV-586 
missing data, IV-597 
model, IV-585 
overview, IV-581 
Quick Graphs, IV-586 
usage, IV-586 
Type I error, IV-21 
Type II error, IV-22 
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u charts, IV-133, IV-134 
unbalanced designs 

in analysis of variance, 11-29 
uncertainty coefficient, 1-234 
unfolding models, IV-3 
uniform kernel, IV-364 


V 
validity, I-87 
variance, 1-307 


of estimates, 1-355 
variance charts, IV-124 
variance component models 
see mixed regression 
variance components 
categorical variables, 11-303 
commands, II-310 
examples, 11-311, 11-315, 11-320, 11-323, TI- 
326, 11-328, 11-334, 11-340 
hypothesis test, 11-306 
model estmation, 11-301 
models, 11-301 
options, 11-304 
overview, II-299 
Quick Graph, 11-310 
usage, II-310 
variance inflation factor, 11-70 
variance of prediction, 1-356 
variance paths 
path analysis, 111-401 
varimax rotation, 1-460, I-464 
variograms, IV-388, IV-401 
model, IV-389 
vector model 
in perceptual mapping, IV-5 
Voronoi polygons, IV-385, IV-397, IV-400 


w 


Wald-Wolfowitz runs test, 111-337 
wave model, IV-391 
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Weibull, III-334 two-sample, IV-48 
Weibull distribution, IV-432 
weighted running smoothing, IV-512 
weights, I-23, I-54, 1-135, 1-179, 1-206, 1-246, I- 
248, 1-323, 1-371, 1-408, 1-469, 1-503, 1-544, II-54, 
11-121, 11-122, 11-202, 11-311, 11-357, 11-399, II- 
441, 11-442, 111-23, III-103, 111-104, 111-137, II- 
194, 111-217, 111-283, 111-339, 111-340, 111-364, III- 
385, 111-413, IV-9, IV-63, IV-104, IV-162, IV- 
244, IV-280, IV-305, IV-325, IV-328, IV-366, IV- 
367, IV-410, IV-449, IV-495, IV-498, IV-547, IV- 
587 
Wilcoxon Signed-Rank test, 111-326 
Wilcoxon test, 111-326 
Wilk’s trace, 1-405 
Wilks’ lambda, 1-405, III-225 
Winter's three-parameter model, IV-524 
Within-Group Testing, 111-241, 111-257 
within-subjects differences 

in analysis of variance, 11-32 
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X charts, IV-129 
X-bar charts, IV-123 
plotting with R charts, IV-129 
plotting with s charts, IV-129 
X-MR charts, IV-149 
control limits, IV-149 
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Yates’ correction, 1-226, 1-233 
y-intercept, II-12 

Young's S-STRESS, III-190 
Yule's Q, 1-228 

Yule's Q coefficient, 1-164 
Yule's Y, 1-228, 1-234 
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Z tests 
z-tests, IV-19 
one-sample, IV-46 


